Page MenuHomeSoftware Heritage

staging: Deploy maven indexer/lister/loader
Closed, ResolvedPublic

Description

When the actual diffs land, making it run in staging should help in lifting papercuts.

Plan:

  • docker declaration should be complete now, make it run within docker
  • T3746#77574, D7023: unstuck the listing scheduling
  • D7052: Land
  • Package new version
  • T3746#78452: Failing with a new issue
  • D7139: Fix that issue
  • package new version v2.6.3
  • Make the lister actually list "maven" (load-maven) tasks to load
  • D7178: Make the scheduler actually schedule load-maven tasks
  • T4105: "Industrialize" maven-index-exporter docker image
  • Develop puppet manifest for the maven stack (indexer, lister, loader)
    • D7527: Deploy node with maven index exporter service which computes the expected lister output (export.fld)
    • D7528: Expose the export files through apache
    • D7538: Update lister service to also manage the list maven
    • D7540: Deploy swh-worker@loader_maven service (in charge of dealing with jar files)
    • Install zfs tooling on that node (to reduce future disk space use [2])
  • D7538: Provision the new node to expose the computation results
  • Ensure the loader and lister maven are registered on the staging scheduler [1]
  • D7542: Provision new node
  • Update inventory
  • Configure zfs partitions
  • Update firewall rules to allow icinga reporting
  • Finally schedule new maven lister instance to consume maven-central and clojars (for now)

[1]

$ swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register | grep -i maven
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven
INFO:swh.scheduler.cli.task_type:Create task type load-maven in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven

[2] export.fld can be quite large (maven-central ~18G), others might be less though
clojars ~60Mib)

Revisions and Commits

rSPRE sysadm-provisioning
D7542
rDENV Development environment
D7167
D7023
rDLSMAVEXP maven-index-exporter
D7509
D7508
D7412
D7412
D7412
D7412
D7412
D7412
rDLDBASE Generic VCS/Package Loader
D7178
rCJSWH Jenkins jobs
D7511
rDLS Listers
D7139
D7052
D7052
rSPSITE puppet-swh-site
Closed
D7561
D7538
D7528
D7527
rDDOC Development documentation
D7054

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@borisbaldassari I've got a new issue in the docker run now [1]

It looks like the listing is sending datetime. It should send iso8601 string imsmw (it's the loader which is in charge of converting the iso string into a datetime).

[1] P1280

It's now fixed and packaged.

I'll have to find or craft a better fld file for the docker environment, with valid
poms that provide valid git repos. Will do asap. Please note however that we could use
one of the many fld files provided at [1], but I'd rather validate the whole run with
a simplistic subset.

Agreed about the simplistic subset, thanks in advance.

@borisbaldassari Did you have the chance to have a look for ^ ?

Hi @ardumont, sorry for the delay, wild week here. And thanks for the iso 8601 fix.

I've updated the .fld file in the docker setup to include 2 more pom entries that lead to one svn scm and one git scm entries. Unfortunately these two URLs are obsolete (repositories have been moved or discontinued), but they're valid scm urls, at least from the lister perspective. Honestly, finding a scm entry is like looking for a needle in a haystack. Finding a *valid* scm entry is next to impossible -- or that's my feeling after trying for a couple of hours.

I've submitted D7167 with the new export file, and the origins I get with it are the following:

(swh) boris@debian:docker$ docker-compose exec swh-scheduler bash -c   'psql swh-scheduler -c "select url,visit_type from listed_origins"'
                                                     url                                                     | visit_type 
-------------------------------------------------------------------------------------------------------------+------------
 https://repo1.maven.org/maven2/org/objectstyle/jetty-launcher/1.6/jetty-launcher-1.6-sources.jar            | maven
 https://repo1.maven.org/maven2/org/objectstyle/japp/japp-maven-plugin/3.0/japp-maven-plugin-3.0-sources.jar | maven
 http://svn.apache.org/repos/asf/jakarta/cactus/tags/1.8.0-RC3                                               | svn
 git://git.apache.org/atlas.git                                                                              | git
(4 rows)
ardumont updated the task description. (Show Details)

Hi @ardumont ,

I see there is a lot of progress here, nice!
I try to follow the thread as time allows, but if you're stuck please do not hesitate to notify me.

Great job anyway, have a wonderful end of week!

Yes, thx. Will do.

Have an awesome week end as well.

ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)

Configure zfs partitions:

root@maven-exporter0:~# lsblk | grep vdb
vdb                           254:16   0   50G  0 disk
root@maven-exporter0:~# zpool create -f data /dev/vdb
root@maven-exporter0:~# zpool status
  pool: data
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          vdb       ONLINE       0     0     0
errors: No known data errors
root@maven-exporter0:~# lsblk | grep vdb
vdb                           254:16   0   50G  0 disk
├─vdb1                        254:17   0   50G  0 part
└─vdb9                        254:25   0    8M  0 part
root@maven-exporter0:~# zfs list
NAME   USED  AVAIL     REFER  MOUNTPOINT
data   106K  48.0G       24K  /data
root@maven-exporter0:~# zfs create -o mountpoint=/srv/softwareheritage/maven-index-exporter/ \
  -o atime=off \
  -o relatime=on \
  -o compression=zstd \
  data/mvn-idx-work
root@maven-exporter0:~# zfs create -o mountpoint=/var/www/maven_index_exporter \
  -o atime=off \
  -o relatime=on \
  -o compression=zstd \
  data/mvn-idx-publish
root@maven-exporter0:~# zfs list
NAME                   USED  AVAIL     REFER  MOUNTPOINT
data                   236K  48.0G       24K  /data
data/mvn-idx-publish    24K  48.0G       24K  /var/www/maven_index_exporter
data/mvn-idx-work       24K  48.0G       24K  /srv/softwareheritage/maven-index-exporter/

Trigger the run for clojars:

root@maven-exporter0:~# systemctl start maven_index_exporter@clojars
root@maven-exporter0:~# systemctl status maven_index_exporter@clojars
● maven_index_exporter@clojars.service - Software Heritage Maven Index Exporter clojars
     Loaded: loaded (/etc/systemd/system/maven_index_exporter@.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/maven_index_exporter@clojars.service.d
             └─parameters.conf
     Active: active (running) since Mon 2022-04-11 16:53:51 UTC; 2s ago
TriggeredBy: ● maven_index_exporter@clojars.timer
   Main PID: 4569 (bash)
      Tasks: 9 (limit: 4675)
     Memory: 56.4M
        CPU: 160ms
     CGroup: /system.slice/system-maven_index_exporter.slice/maven_index_exporter@clojars.service
             ├─4569 bash /usr/local/bin/run_maven_index_exporter.sh clojars
             └─4571 docker run -v /srv/softwareheritage/maven-index-exporter//clojars/work:/work -v /var/www/maven_index_exporter:/publish -e MVN_IDX_EXPORTER_BASE_URL=http://clojars.org/repo/ softwareheritage/maven-index-exporter:v0.2.0
...
root@maven-exporter0:~# ls -lah /var/www/maven_index_exporter/export-clojars.fld
-rwxrwxrwx 1 root root 61M Apr 11 16:54 /var/www/maven_index_exporter/export-clojars.fld
root@maven-exporter0:~# zfs get all | grep compress
data                  compressratio         8.26x                                        -
data                  compression           off                                          default
data                  refcompressratio      1.00x                                        -
data/mvn-idx-publish  compressratio         18.88x                                       -
data/mvn-idx-publish  compression           zstd                                         local
data/mvn-idx-publish  refcompressratio      18.88x                                       -
data/mvn-idx-work     compressratio         5.84x                                        -
data/mvn-idx-work     compression           zstd                                         local
data/mvn-idx-work     refcompressratio      5.84x                                        -

# read out of a staging workers:
ardumont@worker0:~% curl -s https://maven-exporter.internal.staging.swh.network/export-clojars.fld | head
doc 0
  field 0
    name u
    type string
    value lein-html5-docs|lein-html5-docs|2.2.0|NA|pom
  field 1
    name m
    type string
    value 1649610414935
  field 2

Trigger the run for maven-central triggered issue due to the high volume of data for that one.
I'll debug some more tomorrow.

Trigger the run for maven-central triggered issue due to the high volume of data for that one.
I'll debug some more tomorrow.

The issue was identified and was a bit of a mix with puppet actually triggering refresh in the middle of service run.
It was a somehow long investigation which got finally fixed by @vsellier in D7561.

So that part is now ok.
No longer is there some puppet interfering in the service run for no good reason.

There are a bunch of maven repositories crawled:

  • clojars
  • maven-central
  • sonatype
  • jbsoss
  • atlassian-public

Next step is to finally schedule some lister instance for at least 2 of those.
I'm aiming at clojars and maven-central for now.

Scheduling and the lister kicked in [1] but that fails on 404 [2].
And that stopped the listing.

So an improvment here would be to continue listing even if some data is not found.

[1]

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-maven-incremental url=http://clojars.org/repo/ index_url=https://maven-exporter.internal.staging.swh.network/export-clojars.fld
Created 1 tasks

Task 30732884
  Next run: today (2022-04-13T14:45:16.340443+00:00)
  Interval: 1 day, 0:00:00
  Type: list-maven-incremental
  Policy: recurring
  Args:
  Keyword args:
    index_url: 'https://maven-exporter.internal.staging.swh.network/export-clojars.fld'
    url: 'http://clojars.org/repo/'

[2] https://sentry.softwareheritage.org/share/issue/e2da55065b524c568c7a442d653b40c6/

Maven central scheduled as well:

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-maven-incremental url=https://repo1.maven.org/maven2/ index_url=https://maven-exporter.internal.staging.swh.network/export-maven-central.fld
Created 1 tasks

Task 30733021
  Next run: today (2022-04-13T14:58:47.772248+00:00)
  Interval: 1 day, 0:00:00
  Type: list-maven-incremental
  Policy: recurring
  Args:
  Keyword args:
    index_url: 'https://maven-exporter.internal.staging.swh.network/export-maven-central.fld'
    url: 'https://repo1.maven.org/maven2/'

[2]

17:00:10 swh-scheduler@db1:5432=> select * from task where type='list-maven-incremental';
+-[ RECORD 1 ]-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id               | 30732884                                                                                                                                                        |
| type             | list-maven-incremental                                                                                                                                          |
| arguments        | {"args": [], "kwargs": {"url": "http://clojars.org/repo/", "index_url": "https://maven-exporter.internal.staging.swh.network/export-clojars.fld"}}              |
| next_run         | 2022-04-14 14:45:46.393693+00                                                                                                                                   |
| current_interval | 1 day                                                                                                                                                           |
| status           | next_run_not_scheduled                                                                                                                                          |
| policy           | recurring                                                                                                                                                       |
| retries_left     | 0                                                                                                                                                               |
| priority         | (null)                                                                                                                                                          |
+-[ RECORD 2 ]-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id               | 30733021                                                                                                                                                        |
| type             | list-maven-incremental                                                                                                                                          |
| arguments        | {"args": [], "kwargs": {"url": "https://repo1.maven.org/maven2/", "index_url": "https://maven-exporter.internal.staging.swh.network/export-maven-central.fld"}} |
| next_run         | 2022-04-13 14:58:47.772248+00                                                                                                                                   |
| current_interval | 1 day                                                                                                                                                           |
| status           | next_run_scheduled                                                                                                                                              |
| policy           | recurring                                                                                                                                                       |
| retries_left     | 0                                                                                                                                                               |
| priority         | (null)                                                                                                                                                          |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+

Time: 5.405 ms

maven central listing is actually ongoing (at least up until the lister founds some 404 and it will behave the same, crash and stop).
Still some origins are now present in listed_origins:

17:04:16 swh-scheduler@db1:5432=> select * from listed_origins where visit_type='maven' limit 10;
+-[ RECORD 1 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-api/1.0.6/util-ee-metadata-war-api-1.0.6-sources.jar                                                                    |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-api", "gid": "org.ow2.util", "time": "2008-05-03T14:56:12+00:00", "version": "1.0.6", "base_url": "https://repo1.maven.org/maven2/"}]}      |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:16.319959+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:16.319959+00                                                                                                                                                            |
| last_update            | 2008-05-03 14:56:12+00                                                                                                                                                                   |
+-[ RECORD 2 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-ws-api/1.0.21/util-ee-metadata-ws-api-1.0.21-sources.jar                                                                    |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-ws-api", "gid": "org.ow2.util", "time": "2009-11-30T08:34:42+00:00", "version": "1.0.21", "base_url": "https://repo1.maven.org/maven2/"}]}      |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:15.114309+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:15.114309+00                                                                                                                                                            |
| last_update            | 2009-11-30 08:34:42+00                                                                                                                                                                   |
+-[ RECORD 3 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-impl/1.0.8/util-ee-metadata-war-impl-1.0.8-sources.jar                                                                  |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-impl", "gid": "org.ow2.util", "time": "2008-07-15T09:25:12+00:00", "version": "1.0.8", "base_url": "https://repo1.maven.org/maven2/"}]}     |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:15.513142+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:15.513142+00                                                                                                                                                            |
| last_update            | 2008-07-15 09:25:12+00                                                                                                                                                                   |
+-[ RECORD 4 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-impl/1.0.10/util-ee-metadata-war-impl-1.0.10-sources.jar                                                                |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-impl", "gid": "org.ow2.util", "time": "2008-10-30T10:08:37+00:00", "version": "1.0.10", "base_url": "https://repo1.maven.org/maven2/"}]}    |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:16.195206+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:16.195206+00                                                                                                                                                            |
| last_update            | 2008-10-30 10:08:37+00                                                                                                                                                                   |
+-[ RECORD 5 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-impl/1.0.28/util-ee-metadata-war-impl-1.0.28-sources.jar                                                                |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-impl", "gid": "org.ow2.util", "time": "2011-03-08T17:09:44+00:00", "version": "1.0.28", "base_url": "https://repo1.maven.org/maven2/"}]}    |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:15.706799+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:15.706799+00                                                                                                                                                            |
| last_update            | 2011-03-08 17:09:44+00                                                                                                                                                                   |
+-[ RECORD 6 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-impl/1.0.24/util-ee-metadata-war-impl-1.0.24-sources.jar                                                                |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-impl", "gid": "org.ow2.util", "time": "2010-10-01T13:21:17+00:00", "version": "1.0.24", "base_url": "https://repo1.maven.org/maven2/"}]}    |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:15.802489+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:15.802489+00                                                                                                                                                            |
| last_update            | 2010-10-01 13:21:17+00                                                                                                                                                                   |
+-[ RECORD 7 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-api/1.0.10/util-ee-metadata-war-api-1.0.10-sources.jar                                                                  |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-api", "gid": "org.ow2.util", "time": "2008-10-30T10:08:03+00:00", "version": "1.0.10", "base_url": "https://repo1.maven.org/maven2/"}]}     |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:17.176812+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:17.176812+00                                                                                                                                                            |
| last_update            | 2008-10-30 10:08:03+00                                                                                                                                                                   |
+-[ RECORD 8 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-ejbjar-impl/1.0.27/util-ee-metadata-ejbjar-impl-1.0.27-sources.jar                                                          |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-ejbjar-impl", "gid": "org.ow2.util", "time": "2011-03-07T11:09:36+00:00", "version": "1.0.27", "base_url": "https://repo1.maven.org/maven2/"}]} |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:17.502304+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:17.502304+00                                                                                                                                                            |
| last_update            | 2011-03-07 11:09:36+00                                                                                                                                                                   |
+-[ RECORD 9 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-ejbjar-api/1.0.8b/util-ee-metadata-ejbjar-api-1.0.8b-sources.jar                                                            |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-ejbjar-api", "gid": "org.ow2.util", "time": "2008-10-06T15:37:43+00:00", "version": "1.0.8b", "base_url": "https://repo1.maven.org/maven2/"}]}  |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:18.090078+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:18.090078+00                                                                                                                                                            |
| last_update            | 2008-10-06 15:37:43+00                                                                                                                                                                   |
+-[ RECORD 10 ]----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-ejbjar-api/1.0.25/util-ee-metadata-ejbjar-api-1.0.25-sources.jar                                                            |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-ejbjar-api", "gid": "org.ow2.util", "time": "2010-11-04T16:02:06+00:00", "version": "1.0.25", "base_url": "https://repo1.maven.org/maven2/"}]}  |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:18.444392+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:18.444392+00                                                                                                                                                            |
| last_update            | 2010-11-04 16:02:06+00                                                                                                                                                                   |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
17:04:31 swh-scheduler@db1:5432=> select now(), count(*) from listed_origins where visit_type='maven';
+------------------------------+-------+
|             now              | count |
+------------------------------+-------+
| 2022-04-13 15:05:41.06688+00 | 10562 |
+------------------------------+-------+
(1 row)

Time: 781.127 ms

The swh-scheduler-scheduler-recurrent service needed a restart to take into account maven tasks to be loaded.

root@scheduler0:~# systemctl restart swh-scheduler-schedule-recurrent.service
root@scheduler0:~# journalctl -xef -u swh-scheduler-schedule-recurrent.service
...
Apr 13 15:07:22 scheduler0 swh[2915556]: INFO:swh.scheduler.celery_backend.recurrent_visits:maven: 5000 visits scheduled in queue swh.loader.package.maven.tasks.LoadMaven
Apr 13 15:07:30 scheduler0 swh[2915556]: INFO:swh.scheduler.celery_backend.recurrent_visits:Skewed fetch for visit type maven with policy already_visited_order_by_lag: fetched 0.0, requested 0.5
Apr 13 15:07:30 scheduler0 swh[2915556]: INFO:swh.scheduler.celery_backend.recurrent_visits:Skewed fetch for visit type maven with policy never_visited_oldest_update_first: fetched 1.0, requested 0.5

So now, it is scheduled... but the maven loader service fails...
Problem of inconsistent lister/loader format:

root@pergamon:~# clush -b -w @staging-workers systemctl status swh-worker@loader_maven | grep raised
Apr 13 15:08:49 worker0 python3[4019789]: [2022-04-13 15:08:49,800: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[ea331279-d1a5-475a-ae28-86278042fd25] raised unexpected: KeyError('url')
Apr 13 15:08:51 worker0 python3[4019789]: [2022-04-13 15:08:51,584: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[528bb4ee-7d5a-4ef2-82ac-65f2fabe376a] raised unexpected: KeyError('url')
Apr 13 15:08:52 worker0 python3[4019789]: [2022-04-13 15:08:52,647: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[e7ad61c5-cf26-43d7-a2b1-c8d3b50f48a8] raised unexpected: KeyError('url')
Apr 13 15:08:59 worker0 python3[4019792]: [2022-04-13 15:08:59,847: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[f60e663e-fc71-4bea-8733-ef948bd0f566] raised unexpected: KeyError('url')
Apr 13 15:09:01 worker0 python3[4019792]: [2022-04-13 15:09:01,923: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[44bb9522-b713-478f-9da0-c9674d96de4e] raised unexpected: KeyError('url')
Apr 13 15:08:51 worker1 python3[786215]: [2022-04-13 15:08:51,928: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[e7f80b15-8a17-4574-b2c8-787e748bbde7] raised unexpected: KeyError('url')
Apr 13 15:08:53 worker1 python3[786215]: [2022-04-13 15:08:53,562: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[fab1f282-28c3-4d40-b640-ffaa3ce8cbff] raised unexpected: KeyError('url')
Apr 13 15:08:57 worker1 python3[786220]: [2022-04-13 15:08:57,036: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[05244497-c2b1-474d-b4f6-ca8dc8670ea8] raised unexpected: KeyError('url')
Apr 13 15:08:58 worker1 python3[786220]: [2022-04-13 15:08:58,885: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[bfa19c5d-a31d-4bd1-8306-59dbf7d4e2f9] raised unexpected: KeyError('url')
Apr 13 15:09:01 worker1 python3[786220]: [2022-04-13 15:09:01,865: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[66853eb9-956f-4692-bfc0-a3fa6155ff4f] raised unexpected: KeyError('url')
Apr 13 15:08:52 worker2 python3[4051889]: [2022-04-13 15:08:52,876: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[60884263-576b-4e49-8bbb-12c51fa152b5] raised unexpected: KeyError('url')
Apr 13 15:08:54 worker2 python3[4051889]: [2022-04-13 15:08:54,365: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[d0cc0283-7064-40a0-81a5-04352aa91087] raised unexpected: KeyError('url')
Apr 13 15:08:56 worker2 python3[4051889]: [2022-04-13 15:08:56,138: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[61892bcf-a14f-499a-b421-8f2721ab339e] raised unexpected: KeyError('url')
Apr 13 15:08:57 worker2 python3[4051889]: [2022-04-13 15:08:57,992: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[6af5221b-be79-4072-9a0b-2f90220d1b38] raised unexpected: KeyError('url')
Apr 13 15:09:01 worker2 python3[4051898]: [2022-04-13 15:09:01,866: ERROR/ForkPoolWorker-9] Task swh.loader.package.maven.tasks.LoadMaven[2134687f-5255-4f3b-ba2b-fefc34ff27f5] raised unexpected: KeyError('url')
Apr 13 15:08:48 worker3 python3[1662315]: [2022-04-13 15:08:48,240: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[58062e98-0a7c-403b-bdb2-ad58d1a2a20a] raised unexpected: KeyError('url')
Apr 13 15:08:51 worker3 python3[1662315]: [2022-04-13 15:08:51,159: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[d2665b21-5d93-45bd-bbaa-c3ca604ab6fa] raised unexpected: KeyError('url')
Apr 13 15:08:56 worker3 python3[1662321]: [2022-04-13 15:08:56,505: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[4daa8907-407c-496d-b2f4-f7456cf4ab24] raised unexpected: KeyError('url')
Apr 13 15:08:58 worker3 python3[1662321]: [2022-04-13 15:08:58,209: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[acb8396d-667b-48d8-9231-1efda5b1a960] raised unexpected: KeyError('url')
Apr 13 15:09:01 worker3 python3[1662321]: [2022-04-13 15:09:01,879: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[379115c6-1360-45b3-80a6-dcbb2f300544] raised unexpected: KeyError('url')

So the gist of the deployment is done, let's fix those lister and loader issue in the dedicated task [1].

[1] T3874

ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.