Page MenuHomeSoftware Heritage

Add support for a kafka-powered replica of the graph.
ClosedPublic

Authored by douardda on Apr 10 2019, 12:24 PM.

Details

Summary

This is in a dedicated compose file so that one must choose to run it (i.e.
it is not a service started by default when using a 'docker-compose up').

It:

  • overrides the swh-storage service to activate the kafka direct writer on swh.journal.objects prefixed topics using thw swh.storage.master ID,
  • overrides the swh-web service to make it use the replica instead of the master storage,
  • starts a db for the replica,
  • starts a storage service based on this db,
  • starts a replayer service that runs the process that listen to kafka to keeps the replica in sync.

Depends on D1392

Diff Detail

Repository
rCDFD Dockerfiles for developers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
README.md
645–652

should use infinitive form for verbs

docker-compose.yml
149–153

Why not override it instead of adding a dep in the default setup?

This revision is now accepted and ready to land.Apr 10 2019, 12:34 PM
olasd added a subscriber: olasd.
olasd added inline comments.
docker-compose.yml
151

Shouldn't that depends be in the override file?

ardumont added inline comments.
README.md
634

mechanism

646

the

This revision was automatically updated to reflect the committed changes.