Page MenuHomeSoftware Heritage

staging: Deploy maven stack fixes
Closed, ResolvedPublic

Description

  • swh.lister v2.8 (includes a new rust lister)
  • swh.loader.core v2.7.2

Plan:

  • Deploy new version of packages
  • Restart swh-worker@lister
  • Reenable and restart swh-worker@loader_maven
  • Trigger back puppet on staging workers
  • Tag grafana
  • Check loader status
  • Restart clojars listing (which is the one failing)

Event Timeline

ardumont changed the task status from Open to Work in Progress.Apr 14 2022, 3:02 PM
ardumont triaged this task as Normal priority.
ardumont created this task.
ardumont moved this task from Backlog to in-progress on the System administration board.

Deploy new version:

root@pergamon:~# clush -b -w @staging-workers "dpkg -l python3-swh.lister python3-swh.loader.core"
---------------
worker[0-3].internal.staging.swh.network (4)
---------------
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                    Version              Architecture Description
+++-=======================-====================-============-=================================================================
ii  python3-swh.lister      2.8.0-1~swh2~bpo10+1 all          Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)
ii  python3-swh.loader.core 2.6.2-1~swh1~bpo10+1 all          Software Heritage Loader Core

ll

ardumont removed a subscriber: borisbaldassari.

Loader maven status are now ingesting without failing:

Apr 14 13:12:23 worker3 python3[1702388]: [2022-04-14 13:12:23,856: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[5a91c4d6-bd6e-42b0-9006-98f11818cf2e] succeeded in 18.872993586584926s: {'status': 'eventful', 'snapshot_id': '43b0c66d089a2cda79c191469bd9d42769f7f340'}
Apr 14 13:12:23 worker3 python3[1702382]: [2022-04-14 13:12:23,863: INFO/MainProcess] Received task: swh.loader.package.maven.tasks.LoadMaven[19cfbd5f-6f6e-4c59-8eef-9069048c1f5f]
Apr 14 13:12:49 worker3 python3[1702388]: [2022-04-14 13:12:49,499: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[400c3bfb-e7da-4abc-b3cd-076cd6355eac] succeeded in 25.635280357673764s: {'status': 'eventful', 'snapshot_id': 'fa670b69ecb064a1fd284fe796c0f37f50a5bb65'}
Apr 14 13:12:49 worker3 python3[1702382]: [2022-04-14 13:12:49,505: INFO/MainProcess] Received task: swh.loader.package.maven.tasks.LoadMaven[6e3a8f4c-9b31-483e-9275-1b52e7a2ac84]
Apr 14 13:13:10 worker3 python3[1702388]: [2022-04-14 13:13:10,789: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[19cfbd5f-6f6e-4c59-8eef-9069048c1f5f] succeeded in 21.282775953412056s: {'status': 'eventful', 'snapshot_id': '22100ce0175da1a9cf316362eafa0a83101b6737'}
Apr 14 13:13:10 worker3 python3[1702382]: [2022-04-14 13:13:10,793: INFO/MainProcess] Received task: swh.loader.package.maven.tasks.LoadMaven[7c9eb013-f94f-41e2-8bc6-db8bfcb64681]
Apr 14 13:13:34 worker3 python3[1702388]: [2022-04-14 13:13:34,201: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[6e3a8f4c-9b31-483e-9275-1b52e7a2ac84] succeeded in 23.40711336955428s: {'status': 'eventful', 'snapshot_id': 'e031accf172dcc1fda57e195f5fcbee5b75d22a3'}
Apr 14 13:13:34 worker3 python3[1702382]: [2022-04-14 13:13:34,206: INFO/MainProcess] Received task: swh.loader.package.maven.tasks.LoadMaven[2ba0fe0f-b2b1-4ad5-bc55-604990b5c52c]
Apr 14 13:14:01 worker3 python3[1702388]: [2022-04-14 13:14:01,091: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[7c9eb013-f94f-41e2-8bc6-db8bfcb64681] succeeded in 26.881736532785s: {'status': 'eventful', 'snapshot_id': 'f32bb2c8a6f03939bb5239012ce69059f793e287'}

Rescheduled the lister instance to scrape clojars and now it continues on, skipping (and not failing) when it fails to retrieve artifacts informations:

Apr 14 13:22:13 worker3 python3[1702327]: [2022-04-14 13:22:13,346: WARNING/ForkPoolWorker-4] POM info page could not be fetched, skipping project 'http://clojars.org/repo/lein-emr/lein-emr/0.1.0-SNAPSHOT/lein-emr-0.1.0-SNAPSHOT.pom'
Apr 14 13:22:20 worker3 python3[1702327]: [2022-04-14 13:22:20,157: WARNING/ForkPoolWorker-4] Unexpected HTTP status code 404 on https://repo.clojars.org/lein-elastic-beanstalk/lein-elastic-beanstalk/0.2.8-SNAPSHOT/lein-elastic-beanstalk-0.2.8-SNAPSHOT.pom: b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>lein-elastic-beanstalk/lein-elastic-beanstalk/0.2.8-SNAPSHOT/lein-elastic-beanstalk-0.2.8-SNAPSHOT.pom</Key><RequestId>1TH5FQET7C5HKR31</RequestId><HostId>wyTBxRxr1cBiCJNbBjlP86WEZqb5tHb+FwQ8lNFcRD/VICvedManD3iIYUsvJG0MVTCWUORGSRhXCr9QTBGw2g==</HostId></Error>'
Apr 14 13:22:20 worker3 python3[1702327]: [2022-04-14 13:22:20,159: WARNING/ForkPoolWorker-4] POM info page could not be fetched, skipping project 'http://clojars.org/repo/lein-elastic-beanstalk/lein-elastic-beanstalk/0.2.8-SNAPSHOT/lein-elastic-beanstalk-0.2.8-SNAPSHOT.pom'
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.