Change Details

Implies: - [x] new docker image release including swh.lister v3.0 - [x] new docker image release including swh.loader.core v4.0 - [ ] Register new task types to the staging scheduler [1] - [ ] new loader and lister to declare Plan: - Use new elastic worker infra [1] ``` swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.svn WARNING:swh.scheduler.cli.task_type:Existing task type load-svn for module swh.loader.svn.tasks has a different backend name than current code version provides (swh.loader.svn.tasks.DumpMountAndLoadSvnRepository vs. swh.loader.svn.tasks.LoadSvnRepository) INFO:swh.scheduler.cli.task_type:Create task type load-svn-from-remote-dump in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.mercurial INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.git INFO:swh.scheduler.cli.task_type:Create task type load-git-from-dir in scheduler INFO:swh.scheduler.cli.task_type:Create task type load-git-from-zip in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.git_disk INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cvs INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.archive INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cran INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.crates INFO:swh.scheduler.cli.task_type:Create task type load-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.debian INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.deposit INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.nixguix INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.npm INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.opam INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.pypi INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.bzr INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.arch INFO:swh.scheduler.cli.task_type:Create task type list-arch in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.aur INFO:swh.scheduler.cli.task_type:Create task type list-aur in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bitbucket INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bower INFO:swh.scheduler.cli.task_type:Create task type list-bower in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cgit INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cran INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.crates INFO:swh.scheduler.cli.task_type:Create task type list-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.debian INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.github INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitlab INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gnu INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.golang INFO:swh.scheduler.cli.task_type:Create task type list-golang in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-golang-incremental in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.npm INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.opam INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.packagist INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.phabricator INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pubdev INFO:swh.scheduler.cli.task_type:Create task type list-pubdev in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pypi INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.sourceforge INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.tuleap INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gogs INFO:swh.scheduler.cli.task_type:Create task type list-gogs-full in scheduler ```

Implies: - [x] new docker image release including swh.lister v3.0 - [x] new docker image release including swh.loader.core v4.0 - [x] Register new task types to the staging scheduler [1] - [x] Missing load-pubdev task type in scheduler -> reason: python3-swh.loader.core v4.0 debian build broken - [ ] Fix debian build ^ as it's needed for the scheduler node to register all new task types - [ ] Schedule one task with type 'list-pubdev' - [ ] archive-staging cluster: declare new loader and lister - [ ] Check listing/loading happening Plan: - Use new elastic worker infra [1] ``` swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.svn WARNING:swh.scheduler.cli.task_type:Existing task type load-svn for module swh.loader.svn.tasks has a different backend name than current code version provides (swh.loader.svn.tasks.DumpMountAndLoadSvnRepository vs. swh.loader.svn.tasks.LoadSvnRepository) INFO:swh.scheduler.cli.task_type:Create task type load-svn-from-remote-dump in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.mercurial INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.git INFO:swh.scheduler.cli.task_type:Create task type load-git-from-dir in scheduler INFO:swh.scheduler.cli.task_type:Create task type load-git-from-zip in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.git_disk INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cvs INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.archive INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cran INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.crates INFO:swh.scheduler.cli.task_type:Create task type load-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.debian INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.deposit INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.nixguix INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.npm INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.opam INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.pypi INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.bzr INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.arch INFO:swh.scheduler.cli.task_type:Create task type list-arch in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.aur INFO:swh.scheduler.cli.task_type:Create task type list-aur in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bitbucket INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bower INFO:swh.scheduler.cli.task_type:Create task type list-bower in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cgit INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cran INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.crates INFO:swh.scheduler.cli.task_type:Create task type list-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.debian INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.github INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitlab INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gnu INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.golang INFO:swh.scheduler.cli.task_type:Create task type list-golang in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-golang-incremental in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.npm INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.opam INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.packagist INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.phabricator INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pubdev INFO:swh.scheduler.cli.task_type:Create task type list-pubdev in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pypi INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.sourceforge INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.tuleap INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gogs INFO:swh.scheduler.cli.task_type:Create task type list-gogs-full in scheduler ```

Implies: - [x] new docker image release including swh.lister v3.0 - [x] new docker image release including swh.loader.core v4.0 - [ - [x] Register new task types to the staging scheduler [1] - [ ] new loader and lister to declarex] Missing load-pubdev task type in scheduler -> reason: python3-swh.loader.core v4.0 debian build broken - [ ] Fix debian build ^ as it's needed for the scheduler node to register all new task types - [ ] Schedule one task with type 'list-pubdev' - [ ] archive-staging cluster: declare new loader and lister - [ ] Check listing/loading happening Plan: - Use new elastic worker infra [1] ``` swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.svn WARNING:swh.scheduler.cli.task_type:Existing task type load-svn for module swh.loader.svn.tasks has a different backend name than current code version provides (swh.loader.svn.tasks.DumpMountAndLoadSvnRepository vs. swh.loader.svn.tasks.LoadSvnRepository) INFO:swh.scheduler.cli.task_type:Create task type load-svn-from-remote-dump in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.mercurial INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.git INFO:swh.scheduler.cli.task_type:Create task type load-git-from-dir in scheduler INFO:swh.scheduler.cli.task_type:Create task type load-git-from-zip in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.git_disk INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cvs INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.archive INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cran INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.crates INFO:swh.scheduler.cli.task_type:Create task type load-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.debian INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.deposit INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.nixguix INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.npm INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.opam INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.pypi INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.bzr INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.arch INFO:swh.scheduler.cli.task_type:Create task type list-arch in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.aur INFO:swh.scheduler.cli.task_type:Create task type list-aur in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bitbucket INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bower INFO:swh.scheduler.cli.task_type:Create task type list-bower in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cgit INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cran INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.crates INFO:swh.scheduler.cli.task_type:Create task type list-crates in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.debian INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.github INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitlab INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gnu INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.golang INFO:swh.scheduler.cli.task_type:Create task type list-golang in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-golang-incremental in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.npm INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.opam INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.packagist INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.phabricator INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pubdev INFO:swh.scheduler.cli.task_type:Create task type list-pubdev in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pypi INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.sourceforge INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.tuleap INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gogs INFO:swh.scheduler.cli.task_type:Create task type list-gogs-full in scheduler ```