Change Details

When the actual diffs land, making it run in staging should help in lifting papercuts. Plan: - [X] docker declaration should be complete now, make it run within docker - [X] T3746#77574, D7023: unstuck the listing scheduling - [X] D7052: Land - [X] Package new version - [X] T3746#78452: Failing with a new issue - [X] D7139: Fix that issue - [X] package new version v2.6.3 - [X] Make the lister actually list "maven" (load-maven) tasks to load - [X] D7178: Make the scheduler actually schedule load-maven tasks - [X] T4105: "Industrialize" maven-index-exporter docker image - [X] Develop puppet manifest for the maven stack (indexer, lister, loader) - [X] D7527: Deploy node with maven index exporter service which computes the expected lister output (export.fld) - [X] D7528: Expose the export files through apache - [X] D7538: Update lister service to also manage the list maven - [X] D7540: Deploy swh-worker@loader_maven service (in charge of dealing with jar files) - [X] Install zfs tooling on that node (to reduce future disk space use [2]) - [X] D7538: Provision the new node to expose the computation results - [X] Ensure the loader and lister maven are registered on the staging scheduler [1] - [ ] Provision new node - [ ] Configure zfs partitions - [ ] Finally schedule new maven lister instance to consume maven-central and clojars (for now) [1] ``` $ swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register | grep -i maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven INFO:swh.scheduler.cli.task_type:Create task type load-maven in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven ``` [2] export.fld can be quite large (maven-central ~18G), others might be less though clojars ~60Mib)

When the actual diffs land, making it run in staging should help in lifting papercuts. Plan: - [X] docker declaration should be complete now, make it run within docker - [X] T3746#77574, D7023: unstuck the listing scheduling - [X] D7052: Land - [X] Package new version - [X] T3746#78452: Failing with a new issue - [X] D7139: Fix that issue - [X] package new version v2.6.3 - [X] Make the lister actually list "maven" (load-maven) tasks to load - [X] D7178: Make the scheduler actually schedule load-maven tasks - [X] T4105: "Industrialize" maven-index-exporter docker image - [X] Develop puppet manifest for the maven stack (indexer, lister, loader) - [X] D7527: Deploy node with maven index exporter service which computes the expected lister output (export.fld) - [X] D7528: Expose the export files through apache - [X] D7538: Update lister service to also manage the list maven - [X] D7540: Deploy swh-worker@loader_maven service (in charge of dealing with jar files) - [X] Install zfs tooling on that node (to reduce future disk space use [2]) - [X] D7538: Provision the new node to expose the computation results - [X] Ensure the loader and lister maven are registered on the staging scheduler [1] - [X] D7542: Provision new node - [X] Update inventory - [X] Configure zfs partitions - [ ] Finally schedule new maven lister instance to consume maven-central and clojars (for now) [1] ``` $ swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register | grep -i maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven INFO:swh.scheduler.cli.task_type:Create task type load-maven in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven ``` [2] export.fld can be quite large (maven-central ~18G), others might be less though clojars ~60Mib)

When the actual diffs land, making it run in staging should help in lifting papercuts. Plan: - [X] docker declaration should be complete now, make it run within docker - [X] T3746#77574, D7023: unstuck the listing scheduling - [X] D7052: Land - [X] Package new version - [X] T3746#78452: Failing with a new issue - [X] D7139: Fix that issue - [X] package new version v2.6.3 - [X] Make the lister actually list "maven" (load-maven) tasks to load - [X] D7178: Make the scheduler actually schedule load-maven tasks - [X] T4105: "Industrialize" maven-index-exporter docker image - [X] Develop puppet manifest for the maven stack (indexer, lister, loader) - [X] D7527: Deploy node with maven index exporter service which computes the expected lister output (export.fld) - [X] D7528: Expose the export files through apache - [X] D7538: Update lister service to also manage the list maven - [X] D7540: Deploy swh-worker@loader_maven service (in charge of dealing with jar files) - [X] Install zfs tooling on that node (to reduce future disk space use [2]) - [X] D7538: Provision the new node to expose the computation results - [X] Ensure the loader and lister maven are registered on the staging scheduler [1] - [ ]- [X] D7542: Provision new node - [ - [X] Update inventory - [X] Configure zfs partitions - [ ] Finally schedule new maven lister instance to consume maven-central and clojars (for now) [1] ``` $ swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register | grep -i maven INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven INFO:swh.scheduler.cli.task_type:Create task type load-maven in scheduler INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven ``` [2] export.fld can be quite large (maven-central ~18G), others might be less though clojars ~60Mib)