Page MenuHomeSoftware Heritage

Cassandra as a storage backend (meta-task)
Open, NormalPublic

Description

Meta task to get my thoughts in order about adding a Cassandra backend to swh-storage. (Started in april)

  • have a draft implementation https://forge.softwareheritage.org/source/swh-storage-cassandra/
  • benchmark to check the performances are not catastrophic https://forge.softwareheritage.org/source/storage-benchmark-deployment/
  • increase test coverage of all behaviors of swh-storage (D 1534 to 1552)
  • numeric origin ids
    • define a replacement T1731
    • get rid of numeric origin ids in all storage clients T1816
      • non-swh-web clients
      • swh-web
        • queries by origin-id D1969
        • paginated queries T1912
    • public API v2 T1805 (postponed)
  • Add the draft Cassandra backend to the docker env
  • Run the draft Cassandra backend with production data
  • Rewrite the Cassandra backend using the experience learned working on the draft
  • Add it to the docker env
  • Write a storage proxy component, that queries the two backends (postgres and cassandra) and compares their results, to check they are the same; and run it in the docker env. This will make sure migrating to Cassandra does not introduce regressions
  • Run it with production data
  • Deploy in production (possibly with the proxy at first)

Event Timeline

vlorentz triaged this task as Normal priority.Jul 8 2019, 1:25 PM
vlorentz created this task.
vlorentz renamed this task from Cassandra storage backend (meta-task) to Cassandra as a storage backend (meta-task).Jul 8 2019, 1:27 PM
vlorentz updated the task description. (Show Details)
vlorentz updated the task description. (Show Details)
vlorentz updated the task description. (Show Details)Jul 8 2019, 3:29 PM
vlorentz updated the task description. (Show Details)Aug 5 2019, 3:21 PM
vlorentz updated the task description. (Show Details)
vlorentz updated the task description. (Show Details)Oct 8 2019, 11:31 AM
vlorentz updated the task description. (Show Details)Oct 8 2019, 11:42 AM
vlorentz updated the task description. (Show Details)Oct 10 2019, 11:59 AM
vlorentz updated the task description. (Show Details)