Page MenuHomeSoftware Heritage

Deploy swh.model 4.1.0 / swh.storage 0.41.0 to staging
Closed, MigratedEdits Locked

Description

Deploy the latest version of swh.model and swh.storage to staging.

  • make sure all new packages are available (swh.model, swh.storage, swh.deposit)
  • run the first half of the database migration, adding new fields
  • stop all workers
  • upgrade and restart gunicorn-swh-storage instances (storage1, webapp)
  • upgrade and restart worker0
  • check that new revisions and releases are getting the new offset fields populated
  • restart all other workers
  • run the data migration for the new fields on old objects
  • add database constraints for all objects once the migration has run

Event Timeline

olasd triaged this task as High priority.Dec 22 2021, 5:08 PM
olasd created this task.
olasd updated the task description. (Show Details)
olasd changed the task status from Open to Work in Progress.Dec 22 2021, 6:00 PM
olasd updated the task description. (Show Details)

All rows have now been updated and the constraints added.

I managed to do the releases update in one go;

To do the revisions update query incrementally, I used the following loop:

for i in `seq 0 4095`; do
    bound=`printf %03x $i`; 
    query="
      update revision
        set 
          date_offset_bytes=_format_offset(date_offset, date_neg_utc_offset),
          committer_date_offset_bytes=_format_offset(committer_date_offset, committer_date_neg_utc_offset)
        where
          id > '\x${bound}0' 
          and id <= '\x${bound}fffffffffffffffffffffffffffffffffffff' 
          and (
            (date is not null and date_offset_bytes is null)
            or (committer_date is not null and committer_date_offset_bytes is null)
          );
    ";
    echo "updating for revisions starting with ${bound}";
    time psql 'host=db1.internal.staging.swh.network port=5432 user=swh dbname=swh' -c "$query";
done

The individual queries started pretty slow (around 3 minutes each), but sped up as the cache on the revision_pkey index warmed up (I assume) to reach around 3s per individual query, so the operation was done overnight. Once that was done, the constraint validation took a few minutes.