Page MenuHomeSoftware Heritage
Create Task
Maniphest T3746

staging: Deploy maven indexer/lister/loader
Closed, MigratedEdits Locked
Actions

Description

When the actual diffs land, making it run in staging should help in lifting papercuts.

Plan:

  • docker declaration should be complete now, make it run within docker
  • T3746#77574, D7023: unstuck the listing scheduling
  • D7052: Land
  • Package new version
  • T3746#78452: Failing with a new issue
  • D7139: Fix that issue
  • package new version v2.6.3
  • Make the lister actually list "maven" (load-maven) tasks to load
  • D7178: Make the scheduler actually schedule load-maven tasks
  • T4105: "Industrialize" maven-index-exporter docker image
  • Develop puppet manifest for the maven stack (indexer, lister, loader)
    • D7527: Deploy node with maven index exporter service which computes the expected lister output (export.fld)
    • D7528: Expose the export files through apache
    • D7538: Update lister service to also manage the list maven
    • D7540: Deploy swh-worker@loader_maven service (in charge of dealing with jar files)
    • Install zfs tooling on that node (to reduce future disk space use [2])
  • D7538: Provision the new node to expose the computation results
  • Ensure the loader and lister maven are registered on the staging scheduler [1]
  • D7542: Provision new node
  • Update inventory
  • Configure zfs partitions
  • Update firewall rules to allow icinga reporting
  • Finally schedule new maven lister instance to consume maven-central and clojars (for now)

[1]

$ swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register | grep -i maven
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven
INFO:swh.scheduler.cli.task_type:Create task type load-maven in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven

[2] export.fld can be quite large (maven-central ~18G), others might be less though
clojars ~60Mib)

Revisions and Commits

rSPRE sysadm-provisioning
D7542rSPREdea7d3c5faf1 staging: Provision new maven-exporter0.internal.staging.swh.network
rDENV Development environment
D7167rDENV547c1d5bb663 maven-docker: update export file to include svn/git entries.
D7023rDENV1f5c1393573b Add missing maven lister task subscription to lister config
rDLSMAVEXP maven-index-exporter
D7509rDLSMAVEXPeb18e1eae7f1 Update tests to check the image behavior
D7508rDLSMAVEXPea0907619805 Make the docker image extract and publish the export.fld directly
D7412rDLSMAVEXP823b87655201 Add --docker-image-update flag to ease update
D7412rDLSMAVEXP07f60b65dc17 run_full_export.py: Fix some more logging instructions
D7412rDLSMAVEXPe3bb2ab67a87 Drop unnecessary instruction in the python script
D7412rDLSMAVEXPad52a674938c docker/Dockerfile: Switch base image to openjdk11:debian-jre
D7412rDLSMAVEXP112daa60834b docs/maven-repositories.md: Fix typos
D7412rDLSMAVEXPdf2240aade29 Adapt run_full_export according to swh cli conventions
rDLDBASE Generic VCS/Package Loader
D7178rDLDBASE937b874030a3 Fix load_maven scheduling task name
rCJSWH Jenkins jobs
D7511rCJSWHc2f56d295ff4 build-docker-image: Introduce a testing stage
rDLS Listers
D7139rDLS7ff13903783a maven: Fix last update datetime
D7052rDLSd4e1e8212ace maven: Fix undef last_update in ListedOrigins.
D7052rDLS24eeabfadec2 maven: dismiss origins if they are malformed - e.g. wrong pom scm format, add…
rSPSITE puppet-swh-site
ClosedD7540 staging: Deploy maven loader
D7561rSPSITEd72e254002ab maven-exporter: fix the systemd reload after each puppet run
D7538rSPSITE7d4baa0d33b8 staging: Deploy maven lister
D7528rSPSITE0d248255f01e staging: Deploy apache service to serve extracted export.fld files
D7527rSPSITEa1a3afb99fd4 staging: Deploy maven index exporter to extract export.fld files
rDDOC Development documentation
D7054rDDOCf653c250ee3a developer-setup: Mention the current mypy version we use

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ardumont added a revision: D7167: maven-docker: update export file to include svn/git entries..Feb 14 2022, 5:37 PM
ardumont added a commit: rDENV547c1d5bb663: maven-docker: update export file to include svn/git entries..Feb 15 2022, 3:07 PM
ardumont added a revision: D7178: Fix load_maven scheduling task name.Feb 15 2022, 4:10 PM
ardumont updated the task description. (Show Details)Feb 15 2022, 4:12 PM
ardumont updated the task description. (Show Details)
ardumont added a commit: rDLDBASE937b874030a3: Fix load_maven scheduling task name.Feb 15 2022, 5:00 PM
ardumont moved this task from in-progress to Backlog on the System administration board.Mar 14 2022, 1:25 PM
ardumont updated the task description. (Show Details)Mar 22 2022, 3:52 PM
ardumont added a revision: D7412: Adapt run_full_export according to swh cli conventions.Mar 22 2022, 6:03 PM
ardumont added a commit: rDLSMAVEXPdf2240aade29: Adapt run_full_export according to swh cli conventions.Mar 24 2022, 10:33 AM
ardumont added a commit: rDLSMAVEXPad52a674938c: docker/Dockerfile: Switch base image to openjdk11:debian-jre.
ardumont added a commit: rDLSMAVEXP112daa60834b: docs/maven-repositories.md: Fix typos.
ardumont added a commit: rDLSMAVEXP823b87655201: Add --docker-image-update flag to ease update.
ardumont added a commit: rDLSMAVEXP07f60b65dc17: run_full_export.py: Fix some more logging instructions.
ardumont added a commit: rDLSMAVEXPe3bb2ab67a87: Drop unnecessary instruction in the python script.
ardumont moved this task from Backlog to Weekly backlog on the System administration board.Mar 24 2022, 2:33 PM
ardumont moved this task from Weekly backlog to in-progress on the System administration board.Mar 24 2022, 3:10 PM
ardumont added a subtask: T4105: Push maven-index-exporter image to docker hub.Apr 1 2022, 4:09 PM
ardumont updated the task description. (Show Details)
ardumont changed the status of subtask T4105: Push maven-index-exporter image to docker hub from Open to Work in Progress.Apr 1 2022, 5:49 PM
borisbaldassari added a comment.Apr 1 2022, 6:10 PM

Hi @ardumont ,

I see there is a lot of progress here, nice!
I try to follow the thread as time allows, but if you're stuck please do not hesitate to notify me.

Great job anyway, have a wonderful end of week!

ardumont added a comment.Apr 1 2022, 6:14 PM

Yes, thx. Will do.

Have an awesome week end as well.

ardumont updated the task description. (Show Details)Apr 1 2022, 6:15 PM
ardumont closed subtask T4105: Push maven-index-exporter image to docker hub as Resolved.Apr 1 2022, 6:26 PM
ardumont updated the task description. (Show Details)Apr 5 2022, 10:36 AM
ardumont added a revision: D7508: Make the docker image extract and publish the export.fld directly.Apr 5 2022, 6:53 PM
ardumont added a revision: D7509: maven-index-exporter: Update tests to check the image behavior.
ardumont added a revision: D7511: build-docker-image: Introduce a testing stage.Apr 6 2022, 9:45 AM
ardumont added a commit: rCJSWHc2f56d295ff4: build-docker-image: Introduce a testing stage.Apr 6 2022, 2:59 PM
ardumont added a commit: rDLSMAVEXPea0907619805: Make the docker image extract and publish the export.fld directly.
ardumont added a commit: rDLSMAVEXPeb18e1eae7f1: Update tests to check the image behavior.
ardumont mentioned this in rCJSWH59b463da5e9f: build-docker-image: Make sure clean up happen.Apr 6 2022, 3:27 PM
ardumont added a revision: D7527: staging: Deploy maven index exporter to extract export.fld files.Apr 7 2022, 5:50 PM
ardumont added a revision: D7528: staging: Deploy apache service to serve extracted export.fld files.Apr 7 2022, 6:12 PM
ardumont updated the task description. (Show Details)Apr 8 2022, 9:38 AM
ardumont mentioned this in rSENV2c6736f6f4af: staging: Add maven index exporter node service.Apr 8 2022, 10:17 AM
ardumont mentioned this in rSENV63ccfbb1c074: vagrant: Generate fake certificate for maven-index-exporter node.
ardumont added a revision: D7538: staging: Deploy maven lister.Apr 8 2022, 2:57 PM
ardumont added a revision: D7540: staging: Deploy maven loader.Apr 8 2022, 3:19 PM
ardumont added a revision: D7542: staging: Provision new maven-exporter0.internal.staging.swh.network.Apr 8 2022, 4:06 PM
ardumont updated the task description. (Show Details)Apr 8 2022, 4:15 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Apr 8 2022, 5:11 PM
ardumont added a commit: rSPSITEa1a3afb99fd4: staging: Deploy maven index exporter to extract export.fld files.Apr 8 2022, 5:12 PM
ardumont added a commit: rSPSITE0d248255f01e: staging: Deploy apache service to serve extracted export.fld files.
ardumont added a commit: rSPSITE7d4baa0d33b8: staging: Deploy maven lister.
ardumont mentioned this in rSPSITE86cfc05050d9: staging: Deploy maven loader.
ardumont mentioned this in rSPSITE1631c55f7ecd: maven-exporter0.staging: Install zfs tools.Apr 8 2022, 6:09 PM
ardumont mentioned this in rSPSITEe541da4648bf: maven-exporter0: Finalize apache deployment.
ardumont updated the task description. (Show Details)Apr 8 2022, 6:11 PM
ardumont updated the task description. (Show Details)Apr 11 2022, 6:52 PM
ardumont updated the task description. (Show Details)Apr 11 2022, 6:55 PM

Configure zfs partitions:

root@maven-exporter0:~# lsblk | grep vdb
vdb                           254:16   0   50G  0 disk
root@maven-exporter0:~# zpool create -f data /dev/vdb
root@maven-exporter0:~# zpool status
  pool: data
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          vdb       ONLINE       0     0     0
errors: No known data errors
root@maven-exporter0:~# lsblk | grep vdb
vdb                           254:16   0   50G  0 disk
├─vdb1                        254:17   0   50G  0 part
└─vdb9                        254:25   0    8M  0 part
root@maven-exporter0:~# zfs list
NAME   USED  AVAIL     REFER  MOUNTPOINT
data   106K  48.0G       24K  /data
root@maven-exporter0:~# zfs create -o mountpoint=/srv/softwareheritage/maven-index-exporter/ \
  -o atime=off \
  -o relatime=on \
  -o compression=zstd \
  data/mvn-idx-work
root@maven-exporter0:~# zfs create -o mountpoint=/var/www/maven_index_exporter \
  -o atime=off \
  -o relatime=on \
  -o compression=zstd \
  data/mvn-idx-publish
root@maven-exporter0:~# zfs list
NAME                   USED  AVAIL     REFER  MOUNTPOINT
data                   236K  48.0G       24K  /data
data/mvn-idx-publish    24K  48.0G       24K  /var/www/maven_index_exporter
data/mvn-idx-work       24K  48.0G       24K  /srv/softwareheritage/maven-index-exporter/
ardumont added a comment.EditedApr 11 2022, 6:57 PM

Trigger the run for clojars:

root@maven-exporter0:~# systemctl start maven_index_exporter@clojars
root@maven-exporter0:~# systemctl status maven_index_exporter@clojars
● maven_index_exporter@clojars.service - Software Heritage Maven Index Exporter clojars
     Loaded: loaded (/etc/systemd/system/maven_index_exporter@.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/maven_index_exporter@clojars.service.d
             └─parameters.conf
     Active: active (running) since Mon 2022-04-11 16:53:51 UTC; 2s ago
TriggeredBy: ● maven_index_exporter@clojars.timer
   Main PID: 4569 (bash)
      Tasks: 9 (limit: 4675)
     Memory: 56.4M
        CPU: 160ms
     CGroup: /system.slice/system-maven_index_exporter.slice/maven_index_exporter@clojars.service
             ├─4569 bash /usr/local/bin/run_maven_index_exporter.sh clojars
             └─4571 docker run -v /srv/softwareheritage/maven-index-exporter//clojars/work:/work -v /var/www/maven_index_exporter:/publish -e MVN_IDX_EXPORTER_BASE_URL=http://clojars.org/repo/ softwareheritage/maven-index-exporter:v0.2.0
...
root@maven-exporter0:~# ls -lah /var/www/maven_index_exporter/export-clojars.fld
-rwxrwxrwx 1 root root 61M Apr 11 16:54 /var/www/maven_index_exporter/export-clojars.fld
root@maven-exporter0:~# zfs get all | grep compress
data                  compressratio         8.26x                                        -
data                  compression           off                                          default
data                  refcompressratio      1.00x                                        -
data/mvn-idx-publish  compressratio         18.88x                                       -
data/mvn-idx-publish  compression           zstd                                         local
data/mvn-idx-publish  refcompressratio      18.88x                                       -
data/mvn-idx-work     compressratio         5.84x                                        -
data/mvn-idx-work     compression           zstd                                         local
data/mvn-idx-work     refcompressratio      5.84x                                        -

# read out of a staging workers:
ardumont@worker0:~% curl -s https://maven-exporter.internal.staging.swh.network/export-clojars.fld | head
doc 0
  field 0
    name u
    type string
    value lein-html5-docs|lein-html5-docs|2.2.0|NA|pom
  field 1
    name m
    type string
    value 1649610414935
  field 2
ardumont updated the task description. (Show Details)Apr 11 2022, 7:01 PM
ardumont added a comment.Apr 11 2022, 7:16 PM

Trigger the run for maven-central triggered issue due to the high volume of data for that one.
I'll debug some more tomorrow.

ardumont mentioned this in rSPSITEcc183901ecf4: maven_index_exporter: Deploy latest v0.2.1.Apr 12 2022, 9:12 AM
ardumont added a commit: rSPREdea7d3c5faf1: staging: Provision new maven-exporter0.internal.staging.swh.network.Apr 12 2022, 9:55 AM
ardumont mentioned this in rSPSITE203ed5ea7424: Add other mainstream public maven repositories.Apr 12 2022, 9:57 AM
ardumont mentioned this in rSPSITE1639dbea845b: maven_index_exporter: Drop trailing /.Apr 12 2022, 11:44 AM
ardumont mentioned this in rSPSITE4e6b8e109097: maven_index_exporter: Remove and name container when done.
ardumont mentioned this in rSPSITE0df535747df8: Exclude /var/lib/docker/zfs volume disk alerts.Apr 12 2022, 11:47 AM
ardumont mentioned this in rSPSITEb2d365b890ac: Exclude some postgres volume disk alerts on met.
ardumont mentioned this in rSPSITEfd487cfa5a57: maven_index_exporter: Deploy latest v0.3.0.Apr 12 2022, 7:21 PM
vsellier added a revision: D7561: maven-exporter: fix the systemd reload after each puppet run.Apr 13 2022, 9:14 AM
vsellier added a commit: rSPSITEd72e254002ab: maven-exporter: fix the systemd reload after each puppet run.Apr 13 2022, 9:48 AM
ardumont added a subscriber: vsellier.Apr 13 2022, 10:08 AM

Trigger the run for maven-central triggered issue due to the high volume of data for that one.
I'll debug some more tomorrow.

The issue was identified and was a bit of a mix with puppet actually triggering refresh in the middle of service run.
It was a somehow long investigation which got finally fixed by @vsellier in D7561.

So that part is now ok.
No longer is there some puppet interfering in the service run for no good reason.

There are a bunch of maven repositories crawled:

  • clojars
  • maven-central
  • sonatype
  • jbsoss
  • atlassian-public

Next step is to finally schedule some lister instance for at least 2 of those.
I'm aiming at clojars and maven-central for now.

ardumont added a comment.EditedApr 13 2022, 4:49 PM

Scheduling and the lister kicked in [1] but that fails on 404 [2].
And that stopped the listing.

So an improvment here would be to continue listing even if some data is not found.

[1]

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-maven-incremental url=http://clojars.org/repo/ index_url=https://maven-exporter.internal.staging.swh.network/export-clojars.fld
Created 1 tasks

Task 30732884
  Next run: today (2022-04-13T14:45:16.340443+00:00)
  Interval: 1 day, 0:00:00
  Type: list-maven-incremental
  Policy: recurring
  Args:
  Keyword args:
    index_url: 'https://maven-exporter.internal.staging.swh.network/export-clojars.fld'
    url: 'http://clojars.org/repo/'

[2] https://sentry.softwareheritage.org/share/issue/e2da55065b524c568c7a442d653b40c6/

ardumont added a comment.Apr 13 2022, 5:01 PM

Maven central scheduled as well:

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-maven-incremental url=https://repo1.maven.org/maven2/ index_url=https://maven-exporter.internal.staging.swh.network/export-maven-central.fld
Created 1 tasks

Task 30733021
  Next run: today (2022-04-13T14:58:47.772248+00:00)
  Interval: 1 day, 0:00:00
  Type: list-maven-incremental
  Policy: recurring
  Args:
  Keyword args:
    index_url: 'https://maven-exporter.internal.staging.swh.network/export-maven-central.fld'
    url: 'https://repo1.maven.org/maven2/'

[2]

17:00:10 swh-scheduler@db1:5432=> select * from task where type='list-maven-incremental';
+-[ RECORD 1 ]-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id               | 30732884                                                                                                                                                        |
| type             | list-maven-incremental                                                                                                                                          |
| arguments        | {"args": [], "kwargs": {"url": "http://clojars.org/repo/", "index_url": "https://maven-exporter.internal.staging.swh.network/export-clojars.fld"}}              |
| next_run         | 2022-04-14 14:45:46.393693+00                                                                                                                                   |
| current_interval | 1 day                                                                                                                                                           |
| status           | next_run_not_scheduled                                                                                                                                          |
| policy           | recurring                                                                                                                                                       |
| retries_left     | 0                                                                                                                                                               |
| priority         | (null)                                                                                                                                                          |
+-[ RECORD 2 ]-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id               | 30733021                                                                                                                                                        |
| type             | list-maven-incremental                                                                                                                                          |
| arguments        | {"args": [], "kwargs": {"url": "https://repo1.maven.org/maven2/", "index_url": "https://maven-exporter.internal.staging.swh.network/export-maven-central.fld"}} |
| next_run         | 2022-04-13 14:58:47.772248+00                                                                                                                                   |
| current_interval | 1 day                                                                                                                                                           |
| status           | next_run_scheduled                                                                                                                                              |
| policy           | recurring                                                                                                                                                       |
| retries_left     | 0                                                                                                                                                               |
| priority         | (null)                                                                                                                                                          |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+

Time: 5.405 ms
ardumont added a comment.Apr 13 2022, 5:05 PM

maven central listing is actually ongoing (at least up until the lister founds some 404 and it will behave the same, crash and stop).
Still some origins are now present in listed_origins:

17:04:16 swh-scheduler@db1:5432=> select * from listed_origins where visit_type='maven' limit 10;
+-[ RECORD 1 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-api/1.0.6/util-ee-metadata-war-api-1.0.6-sources.jar                                                                    |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-api", "gid": "org.ow2.util", "time": "2008-05-03T14:56:12+00:00", "version": "1.0.6", "base_url": "https://repo1.maven.org/maven2/"}]}      |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:16.319959+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:16.319959+00                                                                                                                                                            |
| last_update            | 2008-05-03 14:56:12+00                                                                                                                                                                   |
+-[ RECORD 2 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-ws-api/1.0.21/util-ee-metadata-ws-api-1.0.21-sources.jar                                                                    |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-ws-api", "gid": "org.ow2.util", "time": "2009-11-30T08:34:42+00:00", "version": "1.0.21", "base_url": "https://repo1.maven.org/maven2/"}]}      |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:15.114309+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:15.114309+00                                                                                                                                                            |
| last_update            | 2009-11-30 08:34:42+00                                                                                                                                                                   |
+-[ RECORD 3 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-impl/1.0.8/util-ee-metadata-war-impl-1.0.8-sources.jar                                                                  |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-impl", "gid": "org.ow2.util", "time": "2008-07-15T09:25:12+00:00", "version": "1.0.8", "base_url": "https://repo1.maven.org/maven2/"}]}     |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:15.513142+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:15.513142+00                                                                                                                                                            |
| last_update            | 2008-07-15 09:25:12+00                                                                                                                                                                   |
+-[ RECORD 4 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-impl/1.0.10/util-ee-metadata-war-impl-1.0.10-sources.jar                                                                |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-impl", "gid": "org.ow2.util", "time": "2008-10-30T10:08:37+00:00", "version": "1.0.10", "base_url": "https://repo1.maven.org/maven2/"}]}    |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:16.195206+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:16.195206+00                                                                                                                                                            |
| last_update            | 2008-10-30 10:08:37+00                                                                                                                                                                   |
+-[ RECORD 5 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-impl/1.0.28/util-ee-metadata-war-impl-1.0.28-sources.jar                                                                |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-impl", "gid": "org.ow2.util", "time": "2011-03-08T17:09:44+00:00", "version": "1.0.28", "base_url": "https://repo1.maven.org/maven2/"}]}    |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:15.706799+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:15.706799+00                                                                                                                                                            |
| last_update            | 2011-03-08 17:09:44+00                                                                                                                                                                   |
+-[ RECORD 6 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-impl/1.0.24/util-ee-metadata-war-impl-1.0.24-sources.jar                                                                |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-impl", "gid": "org.ow2.util", "time": "2010-10-01T13:21:17+00:00", "version": "1.0.24", "base_url": "https://repo1.maven.org/maven2/"}]}    |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:15.802489+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:15.802489+00                                                                                                                                                            |
| last_update            | 2010-10-01 13:21:17+00                                                                                                                                                                   |
+-[ RECORD 7 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-war-api/1.0.10/util-ee-metadata-war-api-1.0.10-sources.jar                                                                  |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-war-api", "gid": "org.ow2.util", "time": "2008-10-30T10:08:03+00:00", "version": "1.0.10", "base_url": "https://repo1.maven.org/maven2/"}]}     |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:17.176812+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:17.176812+00                                                                                                                                                            |
| last_update            | 2008-10-30 10:08:03+00                                                                                                                                                                   |
+-[ RECORD 8 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-ejbjar-impl/1.0.27/util-ee-metadata-ejbjar-impl-1.0.27-sources.jar                                                          |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-ejbjar-impl", "gid": "org.ow2.util", "time": "2011-03-07T11:09:36+00:00", "version": "1.0.27", "base_url": "https://repo1.maven.org/maven2/"}]} |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:17.502304+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:17.502304+00                                                                                                                                                            |
| last_update            | 2011-03-07 11:09:36+00                                                                                                                                                                   |
+-[ RECORD 9 ]-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-ejbjar-api/1.0.8b/util-ee-metadata-ejbjar-api-1.0.8b-sources.jar                                                            |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-ejbjar-api", "gid": "org.ow2.util", "time": "2008-10-06T15:37:43+00:00", "version": "1.0.8b", "base_url": "https://repo1.maven.org/maven2/"}]}  |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:18.090078+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:18.090078+00                                                                                                                                                            |
| last_update            | 2008-10-06 15:37:43+00                                                                                                                                                                   |
+-[ RECORD 10 ]----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| lister_id              | ab79de00-a2d3-44ac-90a8-f53c35c5d893                                                                                                                                                     |
| url                    | https://repo1.maven.org/maven2/org/ow2/util/util-ee-metadata-ejbjar-api/1.0.25/util-ee-metadata-ejbjar-api-1.0.25-sources.jar                                                            |
| visit_type             | maven                                                                                                                                                                                    |
| extra_loader_arguments | {"artifacts": [{"aid": "util-ee-metadata-ejbjar-api", "gid": "org.ow2.util", "time": "2010-11-04T16:02:06+00:00", "version": "1.0.25", "base_url": "https://repo1.maven.org/maven2/"}]}  |
| enabled                | t                                                                                                                                                                                        |
| first_seen             | 2022-04-13 15:00:18.444392+00                                                                                                                                                            |
| last_seen              | 2022-04-13 15:00:18.444392+00                                                                                                                                                            |
| last_update            | 2010-11-04 16:02:06+00                                                                                                                                                                   |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
17:04:31 swh-scheduler@db1:5432=> select now(), count(*) from listed_origins where visit_type='maven';
+------------------------------+-------+
|             now              | count |
+------------------------------+-------+
| 2022-04-13 15:05:41.06688+00 | 10562 |
+------------------------------+-------+
(1 row)

Time: 781.127 ms
ardumont added a comment.Apr 13 2022, 5:10 PM

The swh-scheduler-scheduler-recurrent service needed a restart to take into account maven tasks to be loaded.

root@scheduler0:~# systemctl restart swh-scheduler-schedule-recurrent.service
root@scheduler0:~# journalctl -xef -u swh-scheduler-schedule-recurrent.service
...
Apr 13 15:07:22 scheduler0 swh[2915556]: INFO:swh.scheduler.celery_backend.recurrent_visits:maven: 5000 visits scheduled in queue swh.loader.package.maven.tasks.LoadMaven
Apr 13 15:07:30 scheduler0 swh[2915556]: INFO:swh.scheduler.celery_backend.recurrent_visits:Skewed fetch for visit type maven with policy already_visited_order_by_lag: fetched 0.0, requested 0.5
Apr 13 15:07:30 scheduler0 swh[2915556]: INFO:swh.scheduler.celery_backend.recurrent_visits:Skewed fetch for visit type maven with policy never_visited_oldest_update_first: fetched 1.0, requested 0.5

So now, it is scheduled... but the maven loader service fails...
Problem of inconsistent lister/loader format:

root@pergamon:~# clush -b -w @staging-workers systemctl status swh-worker@loader_maven | grep raised
Apr 13 15:08:49 worker0 python3[4019789]: [2022-04-13 15:08:49,800: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[ea331279-d1a5-475a-ae28-86278042fd25] raised unexpected: KeyError('url')
Apr 13 15:08:51 worker0 python3[4019789]: [2022-04-13 15:08:51,584: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[528bb4ee-7d5a-4ef2-82ac-65f2fabe376a] raised unexpected: KeyError('url')
Apr 13 15:08:52 worker0 python3[4019789]: [2022-04-13 15:08:52,647: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[e7ad61c5-cf26-43d7-a2b1-c8d3b50f48a8] raised unexpected: KeyError('url')
Apr 13 15:08:59 worker0 python3[4019792]: [2022-04-13 15:08:59,847: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[f60e663e-fc71-4bea-8733-ef948bd0f566] raised unexpected: KeyError('url')
Apr 13 15:09:01 worker0 python3[4019792]: [2022-04-13 15:09:01,923: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[44bb9522-b713-478f-9da0-c9674d96de4e] raised unexpected: KeyError('url')
Apr 13 15:08:51 worker1 python3[786215]: [2022-04-13 15:08:51,928: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[e7f80b15-8a17-4574-b2c8-787e748bbde7] raised unexpected: KeyError('url')
Apr 13 15:08:53 worker1 python3[786215]: [2022-04-13 15:08:53,562: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[fab1f282-28c3-4d40-b640-ffaa3ce8cbff] raised unexpected: KeyError('url')
Apr 13 15:08:57 worker1 python3[786220]: [2022-04-13 15:08:57,036: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[05244497-c2b1-474d-b4f6-ca8dc8670ea8] raised unexpected: KeyError('url')
Apr 13 15:08:58 worker1 python3[786220]: [2022-04-13 15:08:58,885: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[bfa19c5d-a31d-4bd1-8306-59dbf7d4e2f9] raised unexpected: KeyError('url')
Apr 13 15:09:01 worker1 python3[786220]: [2022-04-13 15:09:01,865: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[66853eb9-956f-4692-bfc0-a3fa6155ff4f] raised unexpected: KeyError('url')
Apr 13 15:08:52 worker2 python3[4051889]: [2022-04-13 15:08:52,876: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[60884263-576b-4e49-8bbb-12c51fa152b5] raised unexpected: KeyError('url')
Apr 13 15:08:54 worker2 python3[4051889]: [2022-04-13 15:08:54,365: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[d0cc0283-7064-40a0-81a5-04352aa91087] raised unexpected: KeyError('url')
Apr 13 15:08:56 worker2 python3[4051889]: [2022-04-13 15:08:56,138: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[61892bcf-a14f-499a-b421-8f2721ab339e] raised unexpected: KeyError('url')
Apr 13 15:08:57 worker2 python3[4051889]: [2022-04-13 15:08:57,992: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[6af5221b-be79-4072-9a0b-2f90220d1b38] raised unexpected: KeyError('url')
Apr 13 15:09:01 worker2 python3[4051898]: [2022-04-13 15:09:01,866: ERROR/ForkPoolWorker-9] Task swh.loader.package.maven.tasks.LoadMaven[2134687f-5255-4f3b-ba2b-fefc34ff27f5] raised unexpected: KeyError('url')
Apr 13 15:08:48 worker3 python3[1662315]: [2022-04-13 15:08:48,240: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[58062e98-0a7c-403b-bdb2-ad58d1a2a20a] raised unexpected: KeyError('url')
Apr 13 15:08:51 worker3 python3[1662315]: [2022-04-13 15:08:51,159: ERROR/ForkPoolWorker-7] Task swh.loader.package.maven.tasks.LoadMaven[d2665b21-5d93-45bd-bbaa-c3ca604ab6fa] raised unexpected: KeyError('url')
Apr 13 15:08:56 worker3 python3[1662321]: [2022-04-13 15:08:56,505: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[4daa8907-407c-496d-b2f4-f7456cf4ab24] raised unexpected: KeyError('url')
Apr 13 15:08:58 worker3 python3[1662321]: [2022-04-13 15:08:58,209: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[acb8396d-667b-48d8-9231-1efda5b1a960] raised unexpected: KeyError('url')
Apr 13 15:09:01 worker3 python3[1662321]: [2022-04-13 15:09:01,879: ERROR/ForkPoolWorker-8] Task swh.loader.package.maven.tasks.LoadMaven[379115c6-1360-45b3-80a6-dcbb2f300544] raised unexpected: KeyError('url')
ardumont added a comment.Apr 13 2022, 5:11 PM

So the gist of the deployment is done, let's fix those lister and loader issue in the dedicated task [1].

[1] T3874

ardumont updated the task description. (Show Details)Apr 13 2022, 5:11 PM
ardumont mentioned this in T3874: staging: Analyze result of the maven listing and ingestion.Apr 13 2022, 5:15 PM
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.Apr 13 2022, 5:17 PM
ardumont closed this task as Resolved.Apr 14 2022, 3:23 PM
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
ardumont changed the status of subtask T3874: staging: Analyze result of the maven listing and ingestion from Open to Work in Progress.Apr 15 2022, 5:12 PM
anlambert added projects: Maven lister, Maven loader.May 13 2022, 4:54 PM
ardumont closed subtask T3874: staging: Analyze result of the maven listing and ingestion as Resolved.Jun 3 2022, 3:18 PM
bchauvet closed subtask T4326: Archive the pom file additionally to the source folder as Invalid.Jun 15 2022, 5:20 PM
ardumont mentioned this in T4330: Deploy maven stack in production.Sep 8 2022, 10:05 AM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:05 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T4105: Push maven-index-exporter image to docker hub from Resolved to Migrated.Oct 19 2022, 6:06 PM
gitlab-migration changed the status of subtask T4326: Archive the pom file additionally to the source folder from Invalid to Migrated.
gitlab-migration changed the status of subtask T3874: staging: Analyze result of the maven listing and ingestion from Resolved to Migrated.Jan 8 2023, 10:03 PM
About · Developer information · Legal