Plan:
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Unknown Object (Maniphest Task) | ||||
| Migrated | gitlab-migration | T1724 Maven Central repository support | ||
| Migrated | gitlab-migration | T3746 staging: Deploy maven indexer/lister/loader | ||
| Migrated | gitlab-migration | T3874 staging: Analyze result of the maven listing and ingestion | ||
| Migrated | gitlab-migration | T4143 staging: Deploy maven stack fixes |
Deploy new version:
root@pergamon:~# clush -b -w @staging-workers "dpkg -l python3-swh.lister python3-swh.loader.core" --------------- worker[0-3].internal.staging.swh.network (4) --------------- Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=======================-====================-============-================================================================= ii python3-swh.lister 2.8.0-1~swh2~bpo10+1 all Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...) ii python3-swh.loader.core 2.6.2-1~swh1~bpo10+1 all Software Heritage Loader Core
ll
Loader maven status are now ingesting without failing:
Apr 14 13:12:23 worker3 python3[1702388]: [2022-04-14 13:12:23,856: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[5a91c4d6-bd6e-42b0-9006-98f11818cf2e] succeeded in 18.872993586584926s: {'status': 'eventful', 'snapshot_id': '43b0c66d089a2cda79c191469bd9d42769f7f340'}
Apr 14 13:12:23 worker3 python3[1702382]: [2022-04-14 13:12:23,863: INFO/MainProcess] Received task: swh.loader.package.maven.tasks.LoadMaven[19cfbd5f-6f6e-4c59-8eef-9069048c1f5f]
Apr 14 13:12:49 worker3 python3[1702388]: [2022-04-14 13:12:49,499: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[400c3bfb-e7da-4abc-b3cd-076cd6355eac] succeeded in 25.635280357673764s: {'status': 'eventful', 'snapshot_id': 'fa670b69ecb064a1fd284fe796c0f37f50a5bb65'}
Apr 14 13:12:49 worker3 python3[1702382]: [2022-04-14 13:12:49,505: INFO/MainProcess] Received task: swh.loader.package.maven.tasks.LoadMaven[6e3a8f4c-9b31-483e-9275-1b52e7a2ac84]
Apr 14 13:13:10 worker3 python3[1702388]: [2022-04-14 13:13:10,789: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[19cfbd5f-6f6e-4c59-8eef-9069048c1f5f] succeeded in 21.282775953412056s: {'status': 'eventful', 'snapshot_id': '22100ce0175da1a9cf316362eafa0a83101b6737'}
Apr 14 13:13:10 worker3 python3[1702382]: [2022-04-14 13:13:10,793: INFO/MainProcess] Received task: swh.loader.package.maven.tasks.LoadMaven[7c9eb013-f94f-41e2-8bc6-db8bfcb64681]
Apr 14 13:13:34 worker3 python3[1702388]: [2022-04-14 13:13:34,201: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[6e3a8f4c-9b31-483e-9275-1b52e7a2ac84] succeeded in 23.40711336955428s: {'status': 'eventful', 'snapshot_id': 'e031accf172dcc1fda57e195f5fcbee5b75d22a3'}
Apr 14 13:13:34 worker3 python3[1702382]: [2022-04-14 13:13:34,206: INFO/MainProcess] Received task: swh.loader.package.maven.tasks.LoadMaven[2ba0fe0f-b2b1-4ad5-bc55-604990b5c52c]
Apr 14 13:14:01 worker3 python3[1702388]: [2022-04-14 13:14:01,091: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[7c9eb013-f94f-41e2-8bc6-db8bfcb64681] succeeded in 26.881736532785s: {'status': 'eventful', 'snapshot_id': 'f32bb2c8a6f03939bb5239012ce69059f793e287'}Rescheduled the lister instance to scrape clojars and now it continues on, skipping (and not failing) when it fails to retrieve artifacts informations:
Apr 14 13:22:13 worker3 python3[1702327]: [2022-04-14 13:22:13,346: WARNING/ForkPoolWorker-4] POM info page could not be fetched, skipping project 'http://clojars.org/repo/lein-emr/lein-emr/0.1.0-SNAPSHOT/lein-emr-0.1.0-SNAPSHOT.pom' Apr 14 13:22:20 worker3 python3[1702327]: [2022-04-14 13:22:20,157: WARNING/ForkPoolWorker-4] Unexpected HTTP status code 404 on https://repo.clojars.org/lein-elastic-beanstalk/lein-elastic-beanstalk/0.2.8-SNAPSHOT/lein-elastic-beanstalk-0.2.8-SNAPSHOT.pom: b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>lein-elastic-beanstalk/lein-elastic-beanstalk/0.2.8-SNAPSHOT/lein-elastic-beanstalk-0.2.8-SNAPSHOT.pom</Key><RequestId>1TH5FQET7C5HKR31</RequestId><HostId>wyTBxRxr1cBiCJNbBjlP86WEZqb5tHb+FwQ8lNFcRD/VICvedManD3iIYUsvJG0MVTCWUORGSRhXCr9QTBGw2g==</HostId></Error>' Apr 14 13:22:20 worker3 python3[1702327]: [2022-04-14 13:22:20,159: WARNING/ForkPoolWorker-4] POM info page could not be fetched, skipping project 'http://clojars.org/repo/lein-elastic-beanstalk/lein-elastic-beanstalk/0.2.8-SNAPSHOT/lein-elastic-beanstalk-0.2.8-SNAPSHOT.pom'