Page MenuHomeSoftware Heritage

swh-journal: The schema migration problem
Closed, MigratedEdits Locked

Description

As per the database, each schema modification begs the question of data migration... (recently the change from occurrence to snapshot, the snapshot naming convention change, the release question, etc...).

We will double this issue for the journal...

Possible solutions:

  • message versioning
  • multiple topics for the same object (+ deprecation date of the old topics)

Event Timeline

ardumont triaged this task as Normal priority.Oct 18 2018, 4:07 PM
ardumont created this task.

Since this "migration problem" also concerns cassandra, maybe an simple approach would be to add a Final version attribute to all model entities (a simple monotonic integer).

vlorentz renamed this task from swh-journal: The migration problem to swh-journal: The schema migration problem.Oct 8 2020, 12:43 PM

I like the Final version attribute idea. But to be clear, this means we will need to add successive versions as extra classes in swh.model.model when we change the schema (which is not necessarily a bad thing), and remove them when we are sure they are not around anymore

douardda claimed this task.

See T3893 instead.