Plan:
- lister v0.15 (includes opam lister)
- loader.core: Tag v0.23 which includes the opam loader
- Register opam listing and loading task in the scheduler [1]
- D6061: Add opam lister task to the worker configuration
- Upgrade loader.core to staging workers and restart swh-worker@lister service
- Trigger first listing [2]
- Monitoring
- D6062: Puppetize new swh-worker@loader_opam
- Deploy new loader
- Start loader service
[1]
swhscheduler@scheduler0:~$ dpkg -l python3-swh.loader.core Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=======================-=====================-============-================================= ii python3-swh.loader.core 0.23.1-1~swh1~bpo10+1 all Software Heritage Loader Core swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register ... INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.opam INFO:swh.scheduler.cli.task_type:Create task type list-opam in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.opam INFO:swh.scheduler.cli.task_type:Create task type load-opam in scheduler ... $ psql service=staging-swh-scheduler 16:49:33 swh-scheduler@db1:5432=> select * from task_type where type like '%opam%'; +-----------+-----------------------------------+----------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+ | type | description | backend_name | default_interval | min_interval | max_interval | backoff_factor | max_queue_length | num_retries | retry_delay | +-----------+-----------------------------------+----------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+ | list-opam | Lister task for the Opam registry | swh.lister.opam.tasks.OpamListerTask | 1 day | 1 day | 1 day | 1 | (null) | (null) | (null) | | load-opam | Load Opam's artifacts | swh.loader.package.opam.tasks.LoadOpam | 1 day | 1 day | 1 day | 1 | (null) | (null) | (null) | +-----------+-----------------------------------+----------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+ (2 rows) Time: 144.033 ms
[2]
swhscheduler@scheduler0:~$ /usr/bin/swh scheduler --config-file scheduler.yml task add list-opam url=https://opam.ocaml.org instance=opam.ocaml.org
Created 1 tasks
Task 24858921
Next run: today (2021-08-05T15:20:39.703665+00:00)
Interval: 1 day, 0:00:00
Type: list-opam
Policy: recurring
Args:
Keyword args:
instance: 'opam.ocaml.org'
url: 'https://opam.ocaml.org'