Page MenuHomeSoftware Heritage
Create Task
Maniphest T3973

Deploy swh.deposit v0.17
Closed, MigratedEdits Locked
Actions

Description

Includes:

  • drop xmltodict dependency in favor of ElementTree implementation
  • deposit: separate deposit types
  • Add warning to various deposit clis when provenance metadata is not provided by clients

Requisite:

  • Tag loader.core (vlorentz did)
  • Fix loader.core debian build [1]
  • deploy loader.core v2.5.2 (upgrade package, then restart swh-worker@loader_deposit)

Plan:

  • Upgrade debian package definition (debian/control) and drop python3-xmltodict dep
  • Tag the deposit
  • Fix deposit debian build [2]
  • Upgrade python3-swh.deposit* package in server (and python3-swh.model)
  • Migrate the db to the latest schema (deposit will be typed either 'code' or 'meta')
  • Restart gunicorn-swh-deposit service
  • pergamon (icinga): Upgrade python3-swh.deposit* dependency for icinga-plugins to do its job ok

[1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/261/console

[2] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DDEP/job/gbp-buildpackage/141/console

Revisions and Commits

rDLDBASE Generic VCS/Package Loader
AbandonedD7256 Ignore quilt .pc/ folder
D7255rDLDBASE5c11fe2ef48b loader/opam/tests: Do not run actual opam init command call
D7248rDLDBASEec6591f6a90f opam: Allow build to run the opam init completely

Related Objects
Search...

Event Timeline

ardumont triaged this task as Normal priority.Feb 24 2022, 10:13 AM
ardumont created this task.
ardumont updated the task description. (Show Details)Feb 24 2022, 10:18 AM
ardumont added a project: System administration.
ardumont added parent tasks: T3971: swh.deposit.errors.ParserError: out of memory: line 1, column 0, T3677: Separate origin-source-code and provenance-metadata in the deposit.
ardumont changed the task status from Open to Work in Progress.Feb 24 2022, 12:07 PM
ardumont moved this task from Backlog to in-progress on the System administration board.
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Feb 24 2022, 2:30 PM
ardumont updated the task description. (Show Details)EditedFeb 24 2022, 3:09 PM
  • Fix deposit debian build

olasd fixed it by adding a Break entry in the debian/control of our python3-pytest backport.
Forcing aptitude to use a python3-pytest-django present in the backport.
That fixed everything.

ardumont updated the task description. (Show Details)Feb 24 2022, 3:15 PM
ardumont updated the task description. (Show Details)Feb 24 2022, 3:22 PM
ardumont added a revision: D7248: opam/tests: Allow build to run the opam init step completely.Feb 24 2022, 3:58 PM
ardumont added a commit: rDLDBASEec6591f6a90f: opam: Allow build to run the opam init completely.Feb 24 2022, 4:02 PM
ardumont added a revision: D7255: loader/opam/tests: Do not run actual opam init command call.Feb 25 2022, 10:09 AM
ardumont added a commit: rDLDBASE5c11fe2ef48b: loader/opam/tests: Do not run actual opam init command call.Feb 25 2022, 10:24 AM
ardumont added a revision: D7256: Ignore quilt .pc/ folder.Feb 25 2022, 12:38 PM
ardumont added a comment.Feb 25 2022, 12:47 PM
  • Fix loader.core debian build [1]

More like worked around it but that will do for now.

After multiple iterations around this (spending way too much time on this again), i've finally worked around
the debian build failure for the loader core.

There is another task to be dealt with (at some point about opam) unrelated to this added in the backlog [2].

[1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/

[2] T3976

Anyway, back to the actual deployment.

ardumont updated the task description. (Show Details)Feb 25 2022, 12:47 PM
ardumont updated the task description. (Show Details)Feb 25 2022, 1:09 PM
ardumont updated the task description. (Show Details)Feb 25 2022, 2:13 PM

After upgrading pergamon and debugging through sentry and cli, the deposit icinga check is back on track.
Triggered back the icinga checks there.

swhworker@pergamon:~$ /usr/bin/swh icinga_plugins --warning 300 --critical 600 check-deposit --server https://deposit.staging.swh.network/1 --username swh --collection swh --password $PASS_DEPOSIT --poll-interval 1 single --archive /usr/share/swh/icinga-plugins/data/deposit/jesuisgpl.tgz --metadata /usr/share/swh/icinga-plugins/data/deposit/jesuisgpl.tgz.xml
DEPOSIT OK - Deposit took 38.89s and succeeded.
| 'load_time' = 33.35s
| 'total_time' = 38.89s
| 'upload_time' = 0.87s
| 'validation_time' = 4.68s
DEPOSIT OK - Deposit Metadata update took 4.00s and succeeded.
| 'total_time' = 42.89s
| 'update_time' = 4.00s
ardumont closed this task as Resolved.Feb 25 2022, 2:16 PM
ardumont claimed this task.
ardumont updated the task description. (Show Details)
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
ardumont moved this task from Backlog to Deployed on the SWORD deposit board.Feb 25 2022, 6:52 PM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:06 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal