Page MenuHomeSoftware Heritage

loader*: Make prepare method resilient to error and origin visit status compliant
Closed, ResolvedPublic

Description

Following the post mortem, i also noticed that the loader-core's prepare method can explode.

Thus not updating correctly the visit status, nor cleaning up correctly.

This is true for all our loaders.

Event Timeline

ardumont created this task.Mar 7 2018, 12:45 PM
ardumont triaged this task as Normal priority.

The gist of this is:

  • separate a prepare_origin_visit method from prepare method
  • prepare_origin_visit is an adapter method to setup origin/visit data (loader dependent because we don't have the same parameter structure...). This could fail (prod issues) but in extreme cases.
  • prepare is a state dependent on the loader's logic, but independent from the origin preparation (this can fail and that's what this issue is all about).

Then adapt accordingly all current loader implementations.

This is now fixed, packaged and deployed.

loader-core commit: 1972979a4248e9155f5a4f72729a0aed43803483

ardumont closed this task as Resolved.Mar 7 2018, 12:51 PM