Page MenuHomeSoftware Heritage

Refactor the backend to make it scale better
Open, HighPublic

Event Timeline

douardda created this task.

I've been thinking a bit about the refactoring of the ProvenanceStorageServer as described in the doc, with a series of queues between the public API and the backend database.

And it's unclear to me how this can be done efficiently. The

splitting the actual db operations into several queues/writers (by id range, for each entity)

is unclear to me. Does it mean there one queue for each entity type? (revision, content, directory) and each relation type? (content_in_revision, etc.) I don't get the split per id range.