Page MenuHomeSoftware Heritage

loader*: Make prepare method resilient to error and origin visit status compliant
Closed, MigratedEdits Locked


Following the post mortem, i also noticed that the loader-core's prepare method can explode.

Thus not updating correctly the visit status, nor cleaning up correctly.

This is true for all our loaders.

Event Timeline

ardumont triaged this task as Normal priority.Mar 7 2018, 12:45 PM
ardumont created this task.

The gist of this is:

  • separate a prepare_origin_visit method from prepare method
  • prepare_origin_visit is an adapter method to setup origin/visit data (loader dependent because we don't have the same parameter structure...). This could fail (prod issues) but in extreme cases.
  • prepare is a state dependent on the loader's logic, but independent from the origin preparation (this can fail and that's what this issue is all about).

Then adapt accordingly all current loader implementations.

This is now fixed, packaged and deployed.

loader-core commit: 1972979a4248e9155f5a4f72729a0aed43803483