Change Details

Implies: - [x] new docker image release including swh.lister v3.0 - [x] new docker image release including swh.loader.core v4.0 - [x] Register new task types to the staging scheduler [1] - [x] Missing load-pubdev task type in scheduler -> reason: python3-swh.loader.core v4.0 debian build broken - [x] Fix debian unstable build ^ as it's needed for the scheduler node to register all new task types [1] - [ ] Fix yet another debian build (buster) [2] - [ ] Schedule one task with type 'list-pubdev' - [ ] archive-staging cluster: declare new loader and lister - [ ] Check listing/loading happening Plan: - Use new elastic worker infra [1] ``` swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.svn WARNING:swh.scheduler.cli.task_type:Existing task type load-svn for module swh.loader.svn.tasks has a different backend name than current code version provides (swh.loader.svn.tasks.DumpMountAndLoadSvnRepository vs. swh.loader.svn.tasks.LoadSvnRepository) INFO:swh.scheduler.cli.task_type:Create task type load-svn-from-remote-dump in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.mercurial INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.git INFO:swh.scheduler.cli.task_type:Create task type load-git-from-dir in scheduler INFO:swh.scheduler.cli.task_type:Create task type load-git-from-zip in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.git_disk INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cvs INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.archive INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cran INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.crates INFO:swh.scheduler.cli.task_type:Create task type load-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.debian INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.deposit INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.nixguix INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.npm INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.opam INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.pypi INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.bzr INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.arch INFO:swh.scheduler.cli.task_type:Create task type list-arch in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.aur INFO:swh.scheduler.cli.task_type:Create task type list-aur in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bitbucket INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bower INFO:swh.scheduler.cli.task_type:Create task type list-bower in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cgit INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cran INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.crates INFO:swh.scheduler.cli.task_type:Create task type list-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.debian INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.github INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitlab INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gnu INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.golang INFO:swh.scheduler.cli.task_type:Create task type list-golang in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-golang-incremental in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.npm INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.opam INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.packagist INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.phabricator INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pubdev INFO:swh.scheduler.cli.task_type:Create task type list-pubdev in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pypi INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.sourceforge INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.tuleap INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gogs INFO:swh.scheduler.cli.task_type:Create task type list-gogs-full in scheduler ``` [1] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/286/console [2] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/288/console

Implies: - [x] new docker image release including swh.lister v3.0 - [x] new docker image release including swh.loader.core v4.0 - [x] Register new task types to the staging scheduler [1] - [x] Missing load-pubdev task type in scheduler -> reason: python3-swh.loader.core v4.0 debian build broken - [x] Fix debian unstable build ^ as it's needed for the scheduler node to register all new task types [1] - [x] Fix yet another debian build (buster) [2] -> False negative (dns issue during build) - [x] Register load-pub task type (among other loader tasks) [1] (amended) - [ ] Schedule one task with type 'list-pubdev' - [ ] archive-staging cluster: declare new loader and lister - [ ] Check listing/loading happening Plan: - Use new elastic worker infra [1] ``` swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register ... INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.svn INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.crates INFO:swh.scheduler.cli.task_type:Create task type load-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.arch INFO:swh.scheduler.cli.task_type:Create task type list-arch in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.aur INFO:swh.scheduler.cli.task_type:Create task type list-aur in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bower INFO:swh.scheduler.cli.task_type:Create task type list-bower in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.crates INFO:swh.scheduler.cli.task_type:Create task type list-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.golang INFO:swh.scheduler.cli.task_type:Create task type list-golang in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-golang-incremental in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pubdev INFO:swh.scheduler.cli.task_type:Create task type list-pubdev in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gogs INFO:swh.scheduler.cli.task_type:Create task type list-gogs-full in scheduler ... # ^^^^ missing couple of loading tasks -> probably an issue with the debian package build swhscheduler@scheduler0:~$ dpkg -l python3-swh.loader.core | grep ii ii python3-swh.loader.core 4.0.0-1~swh2~bpo10+1 all Software Heritage Loader Core swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register | grep pubdev ... INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.arch INFO:swh.scheduler.cli.task_type:Create task type load-arch in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.aur INFO:swh.scheduler.cli.task_type:Create task type load-aur in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.golang INFO:swh.scheduler.cli.task_type:Create task type load-golang in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.pubdev INFO:swh.scheduler.cli.task_type:Create task type load-pubdev in scheduler ... ``` [1] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/286/console [2] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/288/console

Implies: - [x] new docker image release including swh.lister v3.0 - [x] new docker image release including swh.loader.core v4.0 - [x] Register new task types to the staging scheduler [1] - [x] Missing load-pubdev task type in scheduler -> reason: python3-swh.loader.core v4.0 debian build broken - [x] Fix debian unstable build ^ as it's needed for the scheduler node to register all new task types [1] - [ - [x] Fix yet another debian build (buster) [2] -> False negative (dns issue during build) - [x] Register load-pub task type (among other loader tasks) [1] (amended) - [ ] Schedule one task with type 'list-pubdev' - [ ] archive-staging cluster: declare new loader and lister - [ ] Check listing/loading happening Plan: - Use new elastic worker infra [1] ``` swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.svn WARNING:swh.scheduler.cli.task_type:Existing task type load-svn for module swh.loader.svn.tasks has a different backend name than current code version provides (swh.loader.svn.tasks.DumpMountAndLoadSvnRepository vs. swh.loader.svn.tasks.LoadSvnRepository) INFO:swh.scheduler.cli.task_type:Create task type load-svn-from-remote-dump in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.mercurial INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.git INFO:swh.scheduler.cli.task_type:Create task type load-git-from-dir in scheduler INFO:swh.scheduler.cli.task_type:Create task type load-git-from-zip in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.git_disk INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cvs... INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.archive INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cransvn INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.crates INFO:swh.scheduler.cli.task_type:Create task type load-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.debian INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.deposit INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.nixguix INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.npm INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.opam INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.pypi INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.bzr INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.arch INFO:swh.scheduler.cli.task_type:Create task type list-arch in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.aur INFO:swh.scheduler.cli.task_type:Create task type list-aur in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bitbucket INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bower INFO:swh.scheduler.cli.task_type:Create task type list-bower in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cgit INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cran INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.crates INFO:swh.scheduler.cli.task_type:Create task type list-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.debiangolang INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.giteaCreate task type list-golang in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.githubCreate task type list-golang-incremental in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitlabpubdev INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gnuCreate task type list-pubdev in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.golanggogs INFO:swh.scheduler.cli.task_type:Create task type list-golanggs-full in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-golang-incremental in scheduler... INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad# ^^^^ missing couple of loading tasks -> probably an issue with the debian package build swhscheduler@scheduler0:~$ dpkg -l python3-swh.loader.core | grep ii INFO:swh.schedulii python3-swh.loader.cli.task_type:Loading entrypoint for plugin lister.npmore 4.0.0-1~swh2~bpo10+1 all Software Heritage Loader Core INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.opamswhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register | grep pubdev INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.packagist... INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.phabricatorloader.arch INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pubdevCreate task type load-arch in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-pubdev in schedulerLoading entrypoint for plugin loader.aur INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pypiCreate task type load-aur in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.sourceforgeloader.golang INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.tuleapCreate task type load-golang in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.mavenloader.pubdev INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gogsCreate task type load-pubdev in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-gogs-full in scheduler... ``` [1] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/286/console [2] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/288/console