Page MenuHomeSoftware Heritage
Create Task
Maniphest T1600

Write a storage backend that writes to kafka
Closed, MigratedEdits Locked
Actions

Description

a simple a possible, write-only for now

Revisions and Commits

rDSTO Storage manager
ClosedD1294 Add a new JournalWriter interface, which is notified by swh-storage before writing to pgsql.

Related Objects

Event Timeline

douardda created this task.Mar 25 2019, 11:28 AM
zack triaged this task as Normal priority.Mar 25 2019, 3:07 PM
olasd added a subscriber: olasd.Mar 25 2019, 4:44 PM

The intent is to start feeding kafka with some "production-grade" data, in a more reliable fashion that the current PostgreSQL notify mechanism. This will also help us get a better sense of the shape of the data, to make the analysis in T1602 more relevant.

As a first step, we would like to investigate somehow multiplexing writes between PostgreSQL and Kafka.

vlorentz claimed this task.Mar 26 2019, 3:14 PM
vlorentz merged a task: T1606: Make swh-storage write to Kafka ahead of Postgresql.
vlorentz moved this task from Backlog to in progress on the Sprint 2019 03 board.
vlorentz added a subscriber: vlorentz.
vlorentz added a revision: D1294: Add a new JournalWriter interface, which is notified by swh-storage before writing to pgsql..Mar 26 2019, 5:10 PM
vlorentz moved this task from in progress to done on the Sprint 2019 03 board.Apr 10 2019, 10:50 AM
olasd closed this task as Resolved.Jun 18 2019, 2:54 PM
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 9:59 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal