Page MenuHomeSoftware Heritage

Add support for a kafka-powered replica of the graph.
ClosedPublic

Authored by douardda on Apr 10 2019, 12:24 PM.

Details

Summary

This is in a dedicated compose file so that one must choose to run it (i.e.
it is not a service started by default when using a 'docker-compose up').

It:

  • overrides the swh-storage service to activate the kafka direct writer on swh.journal.objects prefixed topics using thw swh.storage.master ID,
  • overrides the swh-web service to make it use the replica instead of the master storage,
  • starts a db for the replica,
  • starts a storage service based on this db,
  • starts a replayer service that runs the process that listen to kafka to keeps the replica in sync.

Depends on D1392

Diff Detail

Repository
rCDFD Dockerfiles for developers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Apr 10 2019, 12:24 PM
vlorentz accepted this revision.Apr 10 2019, 12:34 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
README.md
646–653

should use infinitive form for verbs

docker-compose.yml
149–153

Why not override it instead of adding a dep in the default setup?

This revision is now accepted and ready to land.Apr 10 2019, 12:34 PM
olasd accepted this revision.Apr 10 2019, 12:34 PM
olasd added a subscriber: olasd.
olasd added inline comments.
docker-compose.yml
152

Shouldn't that depends be in the override file?

ardumont added inline comments.
README.md
635

mechanism

647

the

This revision was automatically updated to reflect the committed changes.