Following the post mortem, i also noticed that the loader-core's prepare method can explode.
Thus not updating correctly the visit status, nor cleaning up correctly.
This is true for all our loaders.
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Unknown Object (Maniphest Task) | ||||
| Migrated | gitlab-migration | T367 ingest Google Code repositories | ||
| Migrated | gitlab-migration | T682 Ingest Google Code Mercurial repositories | ||
| Migrated | gitlab-migration | T561 ingest bitbucket (meta task) | ||
| Migrated | gitlab-migration | T593 ingest bitbucket hg/mercurial repositories | ||
| Migrated | gitlab-migration | T329 hg / mercurial loader | ||
| Migrated | gitlab-migration | T964 2018-02-16 worker disk full postmortem | ||
| Migrated | gitlab-migration | T985 loader*: Make prepare method resilient to error and origin visit status compliant |
Event Timeline
Comment Actions
The gist of this is:
- separate a prepare_origin_visit method from prepare method
- prepare_origin_visit is an adapter method to setup origin/visit data (loader dependent because we don't have the same parameter structure...). This could fail (prod issues) but in extreme cases.
- prepare is a state dependent on the loader's logic, but independent from the origin preparation (this can fail and that's what this issue is all about).
Then adapt accordingly all current loader implementations.
This is now fixed, packaged and deployed.
loader-core commit: 1972979a4248e9155f5a4f72729a0aed43803483