Page MenuHomeSoftware Heritage

Cassandra as a storage backend
Open, NormalPublic

Description

Meta task to get my thoughts in order about adding a Cassandra backend to swh-storage. (Started in april)

  • have a draft implementation https://forge.softwareheritage.org/source/swh-storage-cassandra/
  • benchmark to check the performances are not catastrophic https://forge.softwareheritage.org/source/storage-benchmark-deployment/
  • increase test coverage of all behaviors of swh-storage (D 1534 to 1552)
  • numeric origin ids
    • define a replacement T1731
    • get rid of numeric origin ids in all storage clients T1816
      • non-swh-web clients
      • swh-web
        • queries by origin-id D1969
        • paginated queries T1912
    • public API v2 T1805 (postponed)
  • Add the draft Cassandra backend to the docker env
  • Run the draft Cassandra backend with production data
  • Rewrite the Cassandra backend using the experience learned working on the draft
  • Add it to the docker env
  • Write a storage proxy component, that queries the two backends (postgres and cassandra) and compares their results, to check they are the same; and run it in the docker env. This will make sure migrating to Cassandra does not introduce regressions
  • Run it with production data
  • Deploy in production (possibly with the proxy at first)

Related Objects

StatusAssignedTask
OpenNone
Work in Progressvlorentz
Openvlorentz
Resolvedvlorentz
Resolvedvlorentz
Resolvedvlorentz
Resolvedolasd
Resolvedvsellier
Resolvedvsellier
Resolvedvlorentz
Resolvedvlorentz
Resolvedvlorentz
Resolvedvsellier
Resolvedvsellier
Resolvedvlorentz
Resolvedvsellier
Resolvedvsellier
Resolvedvlorentz
Resolvedvsellier
Resolvedardumont
Resolvedardumont
Resolvedvsellier
Resolvedvsellier
Resolvedvsellier
Resolvedvsellier
Resolvedvsellier
Resolvedvsellier
Resolvedanlambert
Resolvedvlorentz
Resolvedvsellier
Resolvedvsellier
Resolvedvsellier
Resolvedvlorentz
Work in Progressvlorentz
OpenNone
Resolvedvlorentz
Resolvedvlorentz
Resolvedvlorentz
Resolvedardumont
Duplicateolasd
Resolvedvlorentz
Resolvedvlorentz
Resolvedvlorentz
Work in Progressvsellier
Resolvedvlorentz
Resolvedvsellier
Resolvedvsellier
Resolvedvsellier
Resolvedvsellier
Duplicatevsellier
Resolvedvsellier
Resolvedvsellier
Work in Progressvsellier
Work in Progressvsellier
Openvlorentz
Wontfixvlorentz
OpenNone

Event Timeline

vlorentz triaged this task as Normal priority.Jul 8 2019, 1:25 PM
vlorentz created this task.
vlorentz renamed this task from Cassandra storage backend (meta-task) to Cassandra as a storage backend (meta-task).Jul 8 2019, 1:27 PM
vlorentz updated the task description. (Show Details)
vlorentz updated the task description. (Show Details)
vlorentz updated the task description. (Show Details)
vlorentz updated the task description. (Show Details)
vlorentz renamed this task from Cassandra as a storage backend (meta-task) to Cassandra as a storage backend.Jan 22 2020, 4:23 PM
vlorentz added a project: meta-task.
vlorentz updated the task description. (Show Details)
vlorentz updated the task description. (Show Details)