Page MenuHomeSoftware Heritage

save code now: Some entries are missing information after update
Closed, MigratedEdits Locked

Description

It so happens that when some replication lag occurs on the replica db (which is used by
the main archive), some information may end up missing in the save code now ui.

It's happening because currently the update routine reads entries to update out of the
save code now statuses considered not final (running, not scheduled, scheduled...). Then
update the save code now entries with information read out of the other backends at this
instant (unbeknownst of the lag as it's an implem detail), refresh the status with what
it got at this instant and never comes back to update what's missing later (when the
replication is caught up). Because it only uses the status of the save code now which
now moved on to a final state (e.g. succeeded, failed).

This needs to be fixed by broadening the scope of the query reading the save code now
entries to update [1], for example using the ones without any visit_date nor any visit
status (even though their status is final). To avoid having too much information to
update, a limit of 1 month back should also put in place.

[1] https://forge.softwareheritage.org/source/swh-web/browse/master/swh/web/common/origin_save.py$613-615

Event Timeline

ardumont triaged this task as Normal priority.May 26 2021, 4:19 PM
ardumont created this task.

Improve the current way of fetching save code now requests to update. Status of those
requests even if updated with their latest status (which is derived out of the
scheduler) will get fetched again for update until the information retrieved out of the
main archive is also configured ("replication lag subsides").

This update landed within the diff ^ above.

Deployed on staging and production.

I also triggered a manual refresh after that to check everything is actually updated
accordingly. It is.

[1]

swhwebapp@webapp1:~$ export DJANGO_SETTINGS_MODULE=swh.web.settings.production
swhwebapp@webapp1:~$ refresh-savecodenow-statuses
Successfully updated 685 save request(s).
swhwebapp@webapp1:~$ refresh-savecodenow-statuses
Successfully updated 1 save request(s).
ardumont claimed this task.