Page MenuHomeSoftware Heritage

Deploy swh.model 4.1.0 / swh.storage 0.41.0 to production
Closed, ResolvedPublic

Description

Deploy the latest version of swh.model and swh.storage to production.

  • make sure all new packages are available (swh.model, swh.storage, swh.deposit)
  • run the first half of the database migration, adding the new fields to both databases (somerset then belvedere)
  • check that postgresql replication still runs properly
  • stop all workers
  • upgrade and restart gunicorn-swh-storage instances
    • saam
    • moma
    • webapp1
    • storage01.euwest.azure
  • upgrade and restart worker08
  • check that new revisions and releases are getting the new offset fields populated
  • restart all other workers
  • run the data migration for the new fields on old objects
  • add database constraints for all objects once the migration has run

Related Objects

StatusAssignedTask
Work in Progressvlorentz
Work in ProgressNone
Work in Progressvlorentz
Resolvedvlorentz
Resolvedvlorentz
Resolvedardumont
Resolvedardumont
Resolvedvlorentz
Resolvedvlorentz
Resolvedardumont
Resolvedardumont
Resolvedvlorentz
Work in Progressvlorentz
Resolvedvlorentz
Resolvedvlorentz
Resolvedvlorentz
OpenNone
Openolasd
Resolvedvlorentz
Resolvedvlorentz
Resolvedolasd

Event Timeline

olasd changed the task status from Open to Work in Progress.Dec 23 2021, 11:45 AM
olasd triaged this task as High priority.
olasd created this task.

I've started stopping most workers, except for a high priority loader on worker01.

All systems upgraded to the new schema.

I've started the data migration process with queries similar to what was written in T3818#76247.

The data migration for the revision table was taking forever (eta was a couple of months); I restarted it with no filtering on ids, monitoring the growth in logical replication spill files...

Well, that wasn't going to work either, so I've now re-launched the migration using id ranges, with 16 parallel queries...

Around half the revision rows have now been migrated to bytes offsets. The migration is still ongoing.

All revisions (supposedly) have been migrated to bytes offsets. I'll wait for the ongoing base backup and vacuum to complete before adding the constraints on the production database.

softwareheritage=#   alter table revision
    add constraint revision_date_offset_not_null
    check (date is null or date_offset_bytes is not null) not valid,
    add constraint revision_committer_date_offset_not_null
    check (committer_date is null or committer_date_offset_bytes is not null) not valid;
ALTER TABLE

softwareheritage=# alter table revision validate constraint revision_date_offset_not_null, validate constraint revision_committer_date_offset_not_null;
ALTER TABLE

All done.

olasd updated the task description. (Show Details)