Page MenuHomeSoftware Heritage

Winery backend: implementation
Open, NormalPublic

Description

At boot time it:

  • Gets an exclusive lock on a Shard in the Write Storage (creating a new one or resuming an existing one)
  • Gets a connection to the Write Storage and the global index
  • Registers for throttling purposes
  • Serves both reads and writes from and to the Read Storage and the Write Storage

When the Shard it is responsible for in the Write Storage is full:

  • It stops accepting requests
  • Writes the Shard to the Read Storage
  • Shutdown

Outcome:

  • Code in the repository tree
  • CI run including integration tests and extensive coverage

Revisions and Commits

Event Timeline

dachary triaged this task as Normal priority.Aug 29 2021, 2:41 PM
dachary created this task.
dachary created this object in space S1 Public.
dachary added a parent task: T3432: Add winery backend.

D6796 is a draft implementation that is not ready for review but can be reviewed anyways in case someone is curious.

  • It only contains the Write Storage side of the logic
  • The logic is assumed to run either the objstorage or the storage workers
  • When a Write Shard is full, it is packed in a separate process running in the background
  • A given objstorage/storage worker will always write into the same Write Shard and will only switch to a new Shard when it is full and scheduled for packing
  • The objstorage/storage worker are assumed to run in an environment that allows them to have write access to the Postgres/Ceph clusters, in the same way they have write permissions to the current Postgres/filesystem