Page MenuHomeSoftware Heritage
Create Task
Maniphest T4214

Production : Deploy bzr/bazaar Loader
Closed, MigratedEdits Locked
Actions

Description

Plan:

  • Tag, Package latest swh.lister v2.9.2 (fix for maven listing) [1]
  • Deploy swh.lister
  • Restart swh-worker@lister
  • Clean up old scheduler entries about wrong listed origins with visit type 'bzr' [2]
  • Trigger full listing for sourceforge instance
  • D7802: Activate bzr loader in production
  • Deploy new swh-worker@loader_bzr
  • Check everything is fine (message consumption, loading, etc...)
    • Services fail to connect properly with rabbitmq (solution: restarting service )
    • No message to consume. Missing the task type registration step on the scheduler. [3]
    • Enjoy workers doing the job [4]

[1] I recalled wrongly it was for bzr so ¯\_(ツ)_/¯

[2]

11:30:26 softwareheritage-scheduler@belvedere:5432=> update listed_origins set enabled=false where visit_type='bzr' and url like 'http://%.sourceforge.net/bzrroot/%';

[3]

11:30:26 softwareheritage-scheduler@belvedere:5432=> update task set status='next_run_not_scheduled', next_run=now() where type='list-sourceforge-full';
UPDATE 1
Time: 993.503 ms
11:33:09 softwareheritage-scheduler@belvedere:5432=> select * from task where type in ('list-sourceforge-full', 'list-sourceforge-incremental');
+-----------+------------------------------+----------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------+
|    id     |             type             |         arguments          |           next_run            | current_interval |         status         |  policy   | retries_left | priority |
+-----------+------------------------------+----------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------+
| 381572107 | list-sourceforge-full        | {"args": [], "kwargs": {}} | 2022-05-10 09:32:07.583825+00 | 90 days          | next_run_scheduled     | recurring |            0 | (null)   |
| 381572406 | list-sourceforge-incremental | {"args": [], "kwargs": {}} | 2022-05-11 13:57:39.486694+00 | 1 day            | next_run_not_scheduled | recurring |            0 | (null)   |
+-----------+------------------------------+----------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------+
(2 rows)

Time: 6.011 ms

[3]

root@saatchi:~#: apt install -y python3-swh.loader.bzr
swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register 2>&1 | grep -i bzr
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.bzr
INFO:swh.scheduler.cli.task_type:Create task type load-bzr in scheduler
root@saatchi:~# journalctl -xef -u swh-scheduler-schedule-recurrent.service | grep -i bazaar
May 10 12:40:21 saatchi swh[1190253]: INFO:swh.scheduler.celery_backend.recurrent_visits:bzr: 4900 visits scheduled in queue swh.loader.bzr.tasks.LoadBazaar

[4]

root@pergamon:~# clush -b -w @swh-workers "systemctl status swh-worker@loader_bzr" | grep -i 'succeeded in'
May 10 12:42:15 worker01 python3[1464594]: [2022-05-10 12:42:15,837: INFO/ForkPoolWorker-1] Task swh.loader.bzr.tasks.LoadBazaar[3c83987f-9bbe-42e4-bd11-d38ce1e43de5] succeeded in 5.385109965223819s: {'status': 'failed'}
May 10 12:43:05 worker01 python3[1464594]: [2022-05-10 12:43:05,941: INFO/ForkPoolWorker-1] Task swh.loader.bzr.tasks.LoadBazaar[973bcf05-eaca-42de-8c70-6f0421fa1936] succeeded in 50.07445917604491s: {'status': 'eventful'}
May 10 12:43:33 worker02 python3[1464376]: [2022-05-10 12:43:33,322: INFO/ForkPoolWorker-3] Task swh.loader.bzr.tasks.LoadBazaar[69f2d3c0-7018-4860-82cd-03c3743a3967] succeeded in 11.869532153941691s: {'status': 'eventful'}
May 10 12:43:35 worker02 python3[1464376]: [2022-05-10 12:43:35,540: INFO/ForkPoolWorker-3] Task swh.loader.bzr.tasks.LoadBazaar[010e629b-150e-4872-a13a-c145d2b0b046] succeeded in 2.1863166890107095s: {'status': 'uneventful'}
May 10 12:41:41 worker03 python3[1466007]: [2022-05-10 12:41:41,581: INFO/ForkPoolWorker-2] Task swh.loader.bzr.tasks.LoadBazaar[abbb544d-f8bf-4385-8597-edc7bcdd7f94] succeeded in 2.4818601550068706s: {'status': 'uneventful'}
May 10 12:42:34 worker03 python3[1466007]: [2022-05-10 12:42:34,905: INFO/ForkPoolWorker-2] Task swh.loader.bzr.tasks.LoadBazaar[26447707-4d2c-460e-9ed1-38152bde09a5] succeeded in 53.30047515081242s: {'status': 'eventful'}
May 10 12:41:31 worker04 python3[1465009]: [2022-05-10 12:41:31,275: INFO/ForkPoolWorker-2] Task swh.loader.bzr.tasks.LoadBazaar[61808b33-41f6-4bb3-8875-42fe9ca4dc4d] succeeded in 11.165508373873308s: {'status': 'failed'}
May 10 12:42:00 worker04 python3[1465009]: [2022-05-10 12:42:00,031: INFO/ForkPoolWorker-2] Task swh.loader.bzr.tasks.LoadBazaar[54004da4-efdb-4a4c-a50b-03718dfc06d6] succeeded in 28.716117864008993s: {'status': 'eventful'}
May 10 12:43:26 worker05 python3[1463981]: [2022-05-10 12:43:26,120: INFO/ForkPoolWorker-4] Task swh.loader.bzr.tasks.LoadBazaar[2d50a834-eda2-4166-8a95-8380f2bedfa4] succeeded in 6.389234246918932s: {'status': 'eventful'}
May 10 12:43:35 worker06 python3[1464279]: [2022-05-10 12:43:35,468: INFO/ForkPoolWorker-3] Task swh.loader.bzr.tasks.LoadBazaar[cd09228e-e64e-4a86-8def-e5a25e7e64eb] succeeded in 3.940549635095522s: {'status': 'eventful'}
May 10 12:42:59 worker07 python3[1460656]: [2022-05-10 12:42:59,446: INFO/ForkPoolWorker-4] Task swh.loader.bzr.tasks.LoadBazaar[9a73406b-3f55-472d-8ee7-a3d682686ca9] succeeded in 2.7027410350274295s: {'status': 'failed'}
May 10 12:43:32 worker09 python3[1470556]: [2022-05-10 12:43:32,837: INFO/ForkPoolWorker-2] Task swh.loader.bzr.tasks.LoadBazaar[c7771267-93ca-4fd0-bd88-17d54ec4af0e] succeeded in 6.112410163972527s: {'status': 'failed'}
May 10 12:43:38 worker09 python3[1470556]: [2022-05-10 12:43:38,999: INFO/ForkPoolWorker-2] Task swh.loader.bzr.tasks.LoadBazaar[a1383332-7fce-4852-90a4-8974465004d3] succeeded in 6.153206372866407s: {'status': 'eventful'}
May 10 12:41:48 worker10 python3[1473357]: [2022-05-10 12:41:48,970: INFO/ForkPoolWorker-1] Task swh.loader.bzr.tasks.LoadBazaar[3bb3abfc-1e33-4846-9012-f1ae45363d8a] succeeded in 6.497761139879003s: {'status': 'eventful'}
May 10 12:41:57 worker11 python3[1471355]: [2022-05-10 12:41:57,765: INFO/ForkPoolWorker-1] Task swh.loader.bzr.tasks.LoadBazaar[d3752b94-35fe-488e-87d0-88995d6c540a] succeeded in 18.910123961046338s: {'status': 'failed'}
May 10 12:42:06 worker12 python3[1470073]: [2022-05-10 12:42:06,225: INFO/ForkPoolWorker-2] Task swh.loader.bzr.tasks.LoadBazaar[6aea27a6-521e-403c-9f70-67b1c6401eb6] succeeded in 3.105594360968098s: {'status': 'failed'}
May 10 12:43:21 worker12 python3[1470073]: [2022-05-10 12:43:21,779: INFO/ForkPoolWorker-2] Task swh.loader.bzr.tasks.LoadBazaar[fb102668-fd49-4889-a9ad-94e741373c1d] succeeded in 75.53467922704294s: {'status': 'eventful'}
May 10 12:43:27 worker13 python3[1460886]: [2022-05-10 12:43:27,294: INFO/ForkPoolWorker-1] Task swh.loader.bzr.tasks.LoadBazaar[3d4b0d60-7c95-477f-a42d-644dc8fd23c5] succeeded in 5.392359251156449s: {'status': 'eventful'}
May 10 12:43:28 worker13 python3[1460886]: [2022-05-10 12:43:28,995: INFO/ForkPoolWorker-1] Task swh.loader.bzr.tasks.LoadBazaar[71bd5982-3393-4329-881a-5431596b9ebf] succeeded in 1.6838052379898727s: {'status': 'uneventful'}
May 10 12:41:28 worker15 python3[1466334]: [2022-05-10 12:41:28,553: INFO/ForkPoolWorker-1] Task swh.loader.bzr.tasks.LoadBazaar[92318756-76ce-4aa1-ac72-ce55e6b518df] succeeded in 1.691385478945449s: {'status': 'failed'}
May 10 12:42:03 worker15 python3[1466334]: [2022-05-10 12:42:03,708: INFO/ForkPoolWorker-1] Task swh.loader.bzr.tasks.LoadBazaar[593bee30-bbf5-4edc-a29c-81a5d751af7e] succeeded in 35.1166769398842s: {'status': 'eventful'}
May 10 12:42:43 worker16 python3[1466466]: [2022-05-10 12:42:43,728: INFO/ForkPoolWorker-1] Task swh.loader.bzr.tasks.LoadBazaar[48b09cd8-4442-447b-a9a4-bdbc53190f60] succeeded in 8.095938676968217s: {'status': 'eventful'}
May 10 12:43:10 worker16 python3[1466466]: [2022-05-10 12:43:10,986: INFO/ForkPoolWorker-1] Task swh.loader.bzr.tasks.LoadBazaar[042c607f-59f2-49e3-a502-0b63742b94d6] succeeded in 27.219813260016963s: {'status': 'eventful'}

Revisions and Commits

rSPSITE puppet-swh-site
D7803rSPSITE5e9a46881ce5 Ensure the loader/lister packages are installed on scheduler node
D7802rSPSITE831dff13410b Deploy bzr loader in production

Related Objects
Search...

Event Timeline

bchauvet triaged this task as Normal priority.Apr 29 2022, 2:34 PM
bchauvet created this task.
ardumont renamed this task from Production : Deploy Bazaar Loader to Production : Deploy bzr/bazaar Loader.May 9 2022, 5:46 PM
ardumont added a parent task: T3610: Bazaar/Breezy loader.
ardumont removed a parent task: Unknown Object (Maniphest Task).
ardumont updated the task description. (Show Details)May 10 2022, 10:58 AM
ardumont updated the task description. (Show Details)
ardumont changed the task status from Open to Work in Progress.May 10 2022, 11:33 AM
ardumont updated the task description. (Show Details)
ardumont moved this task from Backlog to in-progress on the System administration board.
ardumont updated the task description. (Show Details)May 10 2022, 11:45 AM
ardumont added a revision: D7802: Deploy bzr loader in production.May 10 2022, 2:17 PM
ardumont updated the task description. (Show Details)May 10 2022, 2:20 PM
ardumont added a commit: rSPSITE831dff13410b: Deploy bzr loader in production.May 10 2022, 2:23 PM
ardumont mentioned this in rSPSITEb605038f0f6e: worker::instances: Explicit only staging services.
ardumont updated the task description. (Show Details)May 10 2022, 2:27 PM
ardumont updated the task description. (Show Details)May 10 2022, 2:40 PM
ardumont updated the task description. (Show Details)May 10 2022, 2:44 PM
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.
ardumont added a subscriber: ardumont.May 10 2022, 2:54 PM

New triggered listing for sourceforge is doing its job:

14:40:14 softwareheritage-scheduler@belvedere:5432=> select now(), url from listed_origins where visit_type='bzr' and url not like 'https://code.launchpad.net/%' and enabled;
+-------------------------------+---------------------------------------------------------------------+
|              now              |                                 url                                 |
+-------------------------------+---------------------------------------------------------------------+
| 2022-05-10 12:53:41.269355+00 | http://biojazz.bzr.sourceforge.net/bzr/biojazz/trunk                |
| 2022-05-10 12:53:41.269355+00 | http://bugzilla-de.bzr.sourceforge.net/bzr/bugzilla-de              |
| 2022-05-10 12:53:41.269355+00 | http://dbfpy.bzr.sourceforge.net/bzr/dbfpy                          |
| 2022-05-10 12:53:41.269355+00 | http://facile.bzr.sourceforge.net/bzr/facile/trunk                  |
| 2022-05-10 12:53:41.269355+00 | http://hcrond.bzr.sourceforge.net/bzr/hcrond                        |
| 2022-05-10 12:53:41.269355+00 | http://jerichohtml.bzr.sourceforge.net/bzr/jerichohtml              |
| 2022-05-10 12:53:41.269355+00 | http://mh-e.bzr.sourceforge.net/bzr/mh-e                            |
| 2022-05-10 12:53:41.269355+00 | http://ocaml-benchmark.bzr.sourceforge.net/bzr/ocaml-benchmark      |
| 2022-05-10 12:53:41.269355+00 | http://pythonreports.bzr.sourceforge.net/bzr/pythonreports          |
| 2022-05-10 12:53:41.269355+00 | http://qi-tools.bzr.sourceforge.net/bzr/qi-tools/prana_manual       |
| 2022-05-10 12:53:41.269355+00 | http://qi-tools.bzr.sourceforge.net/bzr/qi-tools/release_notes      |
| 2022-05-10 12:53:41.269355+00 | http://r-prj.bzr.sourceforge.net/bzr/r-prj                          |
| 2022-05-10 12:53:41.269355+00 | http://stpiadmin.bzr.sourceforge.net/bzr/stpiadmin/trunk            |
| 2022-05-10 12:53:41.269355+00 | http://ttl.bzr.sourceforge.net/bzr/ttl/net.trajano.eclipse.platform |
| 2022-05-10 12:53:41.269355+00 | http://visualhunter.bzr.sourceforge.net/bzr/visualhunter            |
| 2022-05-10 12:53:41.269355+00 | http://wwh.bzr.sourceforge.net/bzr/wwh                              |
| 2022-05-10 12:53:41.269355+00 | http://xrdua.bzr.sourceforge.net/bzr/xrdua/xrdua                    |
+-------------------------------+---------------------------------------------------------------------+
(17 rows)
ardumont added a revision: D7803: Ensure the loader/lister packages are installed on scheduler node.May 10 2022, 3:11 PM
ardumont added a comment.May 10 2022, 3:18 PM

To avoid infinite growing queue message, i also limited the max queue length:

14:48:26 softwareheritage-scheduler@belvedere:5432=> begin; update task_type set max_queue_length=1000 where max_queue_length is null and type like 'load-%';
BEGIN
Time: 5.700 ms
UPDATE 3
Time: 5.069 ms
14:48:37 *softwareheritage-scheduler@belvedere:5432=> commit;
COMMIT
Time: 5.631 ms

I had been bitten by this in staging (without much of a consequence but still)

ardumont added a commit: rSPSITE5e9a46881ce5: Ensure the loader/lister packages are installed on scheduler node.May 10 2022, 4:03 PM
ardumont added a revision: D7805: Allow setting up logging level in core loader.May 10 2022, 6:03 PM
ardumont removed a revision: D7805: Allow setting up logging level in core loader.May 10 2022, 6:11 PM
ardumont closed this task as Resolved.May 11 2022, 10:20 AM
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:06 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

bchauvet added a parent task: T4079: Extend archive coverage.Nov 3 2022, 10:17 AM
About · Developer information · Legal