We should consider whether we want to backfill the objects missing from the kafka cluster on azure from PostgreSQL.
We'd need to:
As we're going to bootstrap a new kafka cluster in Rocquencourt soon, we should evaluate whether we really need to be doing this beforehand, or if we can just wait to have backfilled the new cluster from scratch instead.
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T2348 swh.journal silently loses large objects instead of rejecting them | ||
| Migrated | gitlab-migration | T2351 Consider backfilling mistakenly rejected large objects from PostgreSQL | ||
| Migrated | gitlab-migration | T2350 Support large messages in swh.journal / kafka |
I've pulled the list of objects from kafka using @seirl's graph export. I'm now looking to make the diff between postgres and that list of objects.
Backfilled objects:
Need double-checking:
Currently sorting the objects that exist in kafka:
TODO:
snapshots, releases, revisions and directories have now been completely backfilled, and no objects of these types are (known to be) missing from the kafka cluster on azure.