Page MenuHomeSoftware Heritage

staging: Deploy latest maven stack
Closed, ResolvedPublic

Description

loader.core: v3.3.0
lister: v2.9.1

Plan:

  • Tag lister and loader.core
  • Stop scheduler-schedule-recurrent-service
  • Clean up '*maven*' queue
  • scheduler: Drop listed_origins entries with visit_type 'maven' [1]
  • Restart scheduler-schedule-recurrent service
  • Upgrade worker deps [2]
  • Restart swh-worker@lister
  • Restart swh-worker@loader_maven
  • Trigger back listing on maven repositories [3]

[1]

15:20:28 swh-scheduler@db1:5432=> delete from listed_origins where visit_type='maven';
DELETE 4048045
Time: 52186.502 ms (00:52.187)

[2]

Setting up python3-swh.lister (2.9.1-1~swh1~bpo10+1) ...
Setting up python3-swh.loader.core (3.3.0-1~swh1~bpo10+1) ...

[3]

15:26:57 swh-scheduler@db1:5432=> update listers set current_state='{}'::jsonb where name='maven';
UPDATE 2
Time: 34.898 ms
15:27:07 swh-scheduler@db1:5432=> select * from listers where name='maven';
+--------------------------------------+-------+-----------------+-------------------------------+---------------+-------------------------------+
|                  id                  | name  |  instance_name  |            created            | current_state |            updated            |
+--------------------------------------+-------+-----------------+-------------------------------+---------------+-------------------------------+
| ab79de00-a2d3-44ac-90a8-f53c35c5d893 | maven | repo1.maven.org | 2022-04-13 14:58:57.054812+00 | {}            | 2022-04-13 14:58:57.054812+00 |
| b1a5a64a-6838-449c-a0ad-89072dd466e0 | maven | clojars.org     | 2022-04-13 14:45:19.786948+00 | {}            | 2022-04-14 15:41:42.958798+00 |
+--------------------------------------+-------+-----------------+-------------------------------+---------------+-------------------------------+
(2 rows)

Time: 7.449 ms

15:27:27 swh-scheduler@db1:5432=> update task set status='next_run_not_scheduled', next_run=now() where type='list-maven-incremental';
UPDATE 2
Time: 96.027 ms

Event Timeline

ardumont changed the task status from Open to Work in Progress.Apr 29 2022, 2:53 PM
ardumont triaged this task as Normal priority.
ardumont created this task.
ardumont updated the task description. (Show Details)
ardumont moved this task from Backlog to in-progress on the System administration board.

New schemed maven origins are getting listed now:

15:35:21 swh-scheduler@db1:5432=> select now(), count(*) from listed_origins where visit_type='maven' limit 10;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-04-29 13:35:23.481962+00 |   879 |
+-------------------------------+-------+
(1 row)

Time: 887.840 ms
15:35:24 swh-scheduler@db1:5432=> select now(), count(*) from listed_origins where visit_type='maven' limit 10;
+------------------------------+-------+
|             now              | count |
+------------------------------+-------+
| 2022-04-29 13:36:40.32933+00 |  1174 |
+------------------------------+-------+
(1 row)

Time: 2070.324 ms (00:02.070)

And ingested:

Apr 29 13:38:51 worker3 python3[215979]: [2022-04-29 13:38:51,458: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[4ac12ab2-7749-4152-baa1-6bf06a587cad] succeeded in 74.83338255400304s: {'status': 'eventful', 'snapshot_id': '571e40dbc59f5f5238d981986211bcabde2d73d8'}
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.