Page MenuHomeSoftware Heritage

Add support for a kafka-powered replica of the graph.
ClosedPublic

Authored by douardda on Apr 10 2019, 12:24 PM.

Details

Summary

This is in a dedicated compose file so that one must choose to run it (i.e.
it is not a service started by default when using a 'docker-compose up').

It:

  • overrides the swh-storage service to activate the kafka direct writer on swh.journal.objects prefixed topics using thw swh.storage.master ID,
  • overrides the swh-web service to make it use the replica instead of the master storage,
  • starts a db for the replica,
  • starts a storage service based on this db,
  • starts a replayer service that runs the process that listen to kafka to keeps the replica in sync.

Depends on D1392

Diff Detail

Repository
rCDFD Dockerfiles for developers
Branch
current
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 5299
Build 7181: arc lint + arc unit

Event Timeline

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
README.md
646–653

should use infinitive form for verbs

docker-compose.yml
149–152

Why not override it instead of adding a dep in the default setup?

This revision is now accepted and ready to land.Apr 10 2019, 12:34 PM
olasd added a subscriber: olasd.
olasd added inline comments.
docker-compose.yml
152

Shouldn't that depends be in the override file?

ardumont added inline comments.
README.md
635

mechanism

647

the

This revision was automatically updated to reflect the committed changes.