Page MenuHomeSoftware Heritage

storage-db: Fixing migration scripts 135-136
ClosedPublic

Authored by ardumont on Jun 15 2019, 8:35 AM.

Details

Summary

Also, the index creations must be done sequentially. Those new indexes need to
be built prior to remove the old index. Otherwise, we break the api (db
answering too slow leads to the api breaking).

136: Drop the extra 'text' mention which fails the alter query

Test Plan

Deploy to production (done)

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont retitled this revision from 135: Dropping production index after new ones replacing it are built to storage-db: Fixing migration scripts 135-136.Jun 15 2019, 8:42 AM
sql/upgrades/136.sql
38 ↗(On Diff #5258)

For information, those kind of query modifying all the table's records cannot really be done that way (well i guess, given a "certain" threshold of data).

With discussing with @olasd, it was done on range with something like (from the db server itself):

i=0; while (( i > 90000 )); do min=$((i * 1000)); max=$(( (i+1) * 1000)); echo "[$min,$max]"; psql -p 5432 -c "update origin_visit set type = origin.type from origin where origin_visit.origin = origin.id and origin_visit.type is null and origin_visit.origin >= $min and origin_visit.origin < $max" softwareheritage; i=$(( i + 1 )); done

Note:

  • we have roughly 90M origins, thus the upper bound mentioned (given a multiple of 1000 which is the interval range used).
  • i did use the pbouncer (that's one more check it works ;)

It started yesterday around 1pm and it got finished this morning (i started 2 kinds of those script, 1 ascending, the other descending).
I had to make an extra pass to ensure there were no more holes (there was, either possibly because i stopped it to improve it with logs, either because the loaders were not completely up-to-date, the client swh-storage package on workers was not for example, i'm not sure).

In any case, migration done btw.

This revision is now accepted and ready to land.Jun 17 2019, 10:05 AM
This revision was automatically updated to reflect the committed changes.