Differential D8531 Diff 30829 swh/lister/puppet/__init__.py

Changeset View

Standalone View

View Options

swh/lister/puppet/init.py

This file was added.

# See the AUTHORS file at the top-level directory of this distribution

# License: GNU General Public License version 3, or any later version

# See top-level LICENSE file for more information

"""

Puppet lister

=============

The Puppet lister list origins from `Puppet Forge`_.

Puppet Forge is a package manager for Puppet modules.

vlorentzUnsubmitted

Done

The Puppet lister list origins from `Puppet Forge`_.

- Puppet Forge is a catalogue of modules created by Puppet, partners, and community

- that helps IT ops practitioners supercharge and simplify their automation processes.

+ Puppet Forge is a package manager for Puppet modules.

As of September 2022 `Puppet Forge`_ list 6917 package names.

Let's keep it simple :)

vlorentz: Let's keep it simple :)

As of September 2022 `Puppet Forge`_ list 6917 package names.

Origins retrieving strategy

---------------------------

To get a list of all package names we call an `http api endpoint`_ which have a

`getModules`_ endpoint.

It returns a paginated list of results and a `next` url.

The api follow `OpenApi 3.0 specifications`.

Page listing

------------

Each page returns a list of ``results`` which are raw data from api response.

The results size is 100 as 100 is the maximum limit size allowed by the api.

vlorentzUnsubmitted

Done

------------

- Each page returns a list of `results` which are raw data from api response.

+ Each page returns a list of ``results`` which are raw data from api response.

The results size is 100 as 100 is the maximum limit size allowed by the api.

Origins from page

vlorentz:

Origins from page

-----------------

The lister yields one hundred origin url per page.

Origin url is the html page corresponding to a package name on the forge, following

this pattern::

"https://forge.puppet.com/modules/{owner}/{pkgname}"

For each origin `last_update`is set via the module "updated_at" value.

As the api also returns all existing versions for a package, we build an `artifacts`

dict in `extra_loader_arguments` with the archive tarball corresponding to each

existing versions.

Example for ``file_concat`` module located at

https://forge.puppet.com/modules/electrical/file_concat::

vlorentzUnsubmitted

Done

existing versions.

- Example for `file_concat` module located at

+ Example for ``file_concat`` module located at

https://forge.puppet.com/modules/electrical/file_concat::

{

"artifacts": {

vlorentz:

{

"artifacts": {

"1.0.0": {

"url": "https://forgeapi.puppet.com/v3/files/electrical-file_concat-1.0.0.tar.gz", # noqa: B950

"version": "1.0.0",

"filename": "electrical-file_concat-1.0.0.tar.gz",

"last_update": "2015-04-09T12:03:13-07:00",

"1.0.1": {

"url": "https://forgeapi.puppet.com/v3/files/electrical-file_concat-1.0.1.tar.gz", # noqa: B950

"version": "1.0.1",

"filename": "electrical-file_concat-1.0.1.tar.gz",

"last_update": "2015-04-17T01:03:46-07:00",

}

Running tests

-------------

Activate the virtualenv and run from within swh-lister directory::

pytest -s -vv --log-cli-level=DEBUG swh/lister/puppet/tests

Testing with Docker

-------------------

Change directory to swh/docker then launch the docker environment::

docker compose up -d

Then schedule a Puppet listing task::

docker compose exec swh-scheduler swh scheduler task add -p oneshot list-puppet

You can follow lister execution by displaying logs of swh-lister service::

docker compose logs -f swh-lister

.. _Puppet Forge: https://forge.puppet.com/

.. _http api endpoint: https://forgeapi.puppet.com/

.. _getModules: https://forgeapi.puppet.com/#tag/Module-Operations/operation/getModules

"""

def register():

from .lister import PuppetLister

return {

"lister": PuppetLister,

"task_modules": ["%s.tasks" % __name__],

}