Implies:
- new docker image release including swh.lister v3.0
- new docker image release including swh.loader.core v4.0
- Register new task types to the staging scheduler [1]
- Missing load-pubdev task type in scheduler -> reason: python3-swh.loader.core v4.0 debian build broken
- Fix debian unstable build ^ as it's needed for the scheduler node to register all new task types [1]
- Fix yet another debian build (buster) [2] -> False negative (dns issue during build)
- Register load-pub task type (among other loader tasks) [1] (amended)
- Schedule one task with type 'list-pubdev' [3]
- D8437: archive-staging cluster: declare new loader and lister
- Checks:
- listing ok [4]
- loading ok
Note:
- Use new elastic worker infra
[1]
swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register ... INFO:swh.scheduler.cli.task_type:Create task type load-crates in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-arch in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-aur in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-bower in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-crates in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-golang in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-golang-incremental in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-pubdev in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-gogs-full in scheduler ... # ^^^^ missing couple of loading tasks -> probably an issue with the debian package build swhscheduler@scheduler0:~$ dpkg -l python3-swh.loader.core | grep ii ii python3-swh.loader.core 4.0.0-1~swh2~bpo10+1 all Software Heritage Loader Core swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register | grep pubdev ... INFO:swh.scheduler.cli.task_type:Create task type load-arch in scheduler INFO:swh.scheduler.cli.task_type:Create task type load-aur in scheduler INFO:swh.scheduler.cli.task_type:Create task type load-golang in scheduler INFO:swh.scheduler.cli.task_type:Create task type load-pubdev in scheduler ...
[3]
swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-pubdev Created 1 tasks Task 33419442 Next run: today (2022-09-09T10:30:00.252081+00:00) Interval: 1 day, 0:00:00 Type: list-pubdev Policy: recurring Args: Keyword args:
[4]
12:37:31 swh-scheduler@db1:5432=> select id, name, instance_name from listers where name='pubdev'; +--------------------------------------+--------+---------------+ | id | name | instance_name | +--------------------------------------+--------+---------------+ | a299a643-0887-44d1-8d71-88da0e1e3b52 | pubdev | pubdev | +--------------------------------------+--------+---------------+ (1 row) Time: 6.813 ms 12:37:29 swh-scheduler@db1:5432=> select now(), visit_type, count(*) from listed_origins where lister_id='a299a643-0887-44d1-8d71-88da0e1e3b52' group by visit_type; +-------------------------------+------------+-------+ | now | visit_type | count | +-------------------------------+------------+-------+ | 2022-09-09 10:37:31.624512+00 | pubdev | 1000 | +-------------------------------+------------+-------+ (1 row) Time: 7.084 ms
[5]
│ loaders [2022-09-09 10:42:40,267: INFO/MainProcess] Connected to amqp://swhconsumer:**@scheduler0.internal.staging.swh.network:5672// │
│ loaders [2022-09-09 10:42:40,823: INFO/MainProcess] loader@loader-pubdev-d699448b7-46dc5 ready. │
│ loaders [2022-09-09 10:44:29,827: INFO/MainProcess] Task swh.loader.package.pubdev.tasks.LoadPubDev[0a0fba7e-97f1-4b6e-b6bc-e4e8190f0897] received │
│ loaders [2022-09-09 10:44:29,872: INFO/MainProcess] Task swh.loader.package.pubdev.tasks.LoadPubDev[388f30f6-6b6f-423f-aa25-f3f5a809d245] received │
│ loaders [2022-09-09 10:44:37,388: INFO/ForkPoolWorker-1] Task swh.loader.package.pubdev.tasks.LoadPubDev[0a0fba7e-97f1-4b6e-b6bc-e4e8190f0897] succeeded in 7.445372300921008s: {'status': 'eventful', 'snapshot_id': '0dab696ae4f51cf95e44 │