Implies:
- [x] new docker image release including swh.lister v3.0
- [x] new docker image release including swh.loader.core v4.0
- [x] Register new task types to the staging scheduler [1]
- [x] Missing load-pubdev task type in scheduler -> reason: python3-swh.loader.core v4.0 debian build broken
- [x] Fix debian unstable build ^ as it's needed for the scheduler node to register all new task types [1]
- [x] Fix yet another debian build (buster) [2] -> False negative (dns issue during build)
- [x] Register load-pub task type (among other loader tasks) [1] (amended)
- [x] Schedule one task with type 'list-pubdev' [3]
- [ ] archive-staging cluster: declare new loader and lister
- [x] Checks:
- [x] listing ok [4]
- [x] loading ok
Plan:
- Use new elastic worker infra
[1]
```
swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register
...
INFO:swh.scheduler.cli.task_type:Create task type load-crates in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-arch in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-aur in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-bower in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-crates in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-golang in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-golang-incremental in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-pubdev in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-gogs-full in scheduler
...
# ^^^^ missing couple of loading tasks -> probably an issue with the debian package build
swhscheduler@scheduler0:~$ dpkg -l python3-swh.loader.core | grep ii
ii python3-swh.loader.core 4.0.0-1~swh2~bpo10+1 all Software Heritage Loader Core
swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register | grep pubdev
...
INFO:swh.scheduler.cli.task_type:Create task type load-arch in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-aur in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-golang in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-pubdev in scheduler
...
```
[1] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/286/console
[2] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/288/console
[3]
```
swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-pubdev
Created 1 tasks
Task 33419442
Next run: today (2022-09-09T10:30:00.252081+00:00)
Interval: 1 day, 0:00:00
Type: list-pubdev
Policy: recurring
Args:
Keyword args:
```
[4]
```
12:37:31 swh-scheduler@db1:5432=> select id, name, instance_name from listers where name='pubdev';
+--------------------------------------+--------+---------------+
| id | name | instance_name |
+--------------------------------------+--------+---------------+
| a299a643-0887-44d1-8d71-88da0e1e3b52 | pubdev | pubdev |
+--------------------------------------+--------+---------------+
(1 row)
Time: 6.813 ms
12:37:29 swh-scheduler@db1:5432=> select now(), visit_type, count(*) from listed_origins where lister_id='a299a643-0887-44d1-8d71-88da0e1e3b52' group by visit_type;
+-------------------------------+------------+-------+
| now | visit_type | count |
+-------------------------------+------------+-------+
| 2022-09-09 10:37:31.624512+00 | pubdev | 1000 |
+-------------------------------+------------+-------+
(1 row)
Time: 7.084 ms
```
[5]
```
│ loaders [2022-09-09 10:42:40,267: INFO/MainProcess] Connected to amqp://swhconsumer:**@scheduler0.internal.staging.swh.network:5672// │
│ loaders [2022-09-09 10:42:40,823: INFO/MainProcess] loader@loader-pubdev-d699448b7-46dc5 ready. │
│ loaders [2022-09-09 10:44:29,827: INFO/MainProcess] Task swh.loader.package.pubdev.tasks.LoadPubDev[0a0fba7e-97f1-4b6e-b6bc-e4e8190f0897] received │
│ loaders [2022-09-09 10:44:29,872: INFO/MainProcess] Task swh.loader.package.pubdev.tasks.LoadPubDev[388f30f6-6b6f-423f-aa25-f3f5a809d245] received │
│ loaders [2022-09-09 10:44:37,388: INFO/ForkPoolWorker-1] Task swh.loader.package.pubdev.tasks.LoadPubDev[0a0fba7e-97f1-4b6e-b6bc-e4e8190f0897] succeeded in 7.445372300921008s: {'status': 'eventful', 'snapshot_id': '0dab696ae4f51cf95e44 │
```