HomeSoftware Heritage

Add support for a kafka-powered replica of the graph.

Description

Add support for a kafka-powered replica of the graph.

This is in a dedicated compose file so that one must choose to run it (i.e.
it is not a service started by default when using a 'docker-compose up').

It:

  • overrides the swh-storage service to activate the kafka direct writer on swh.journal.objects prefixed topics using thw swh.storage.master ID,
  • overrides the swh-web service to make it use the replica instead of the master storage,
  • starts a db for the replica,
  • starts a storage service based on this db,
  • starts a replayer service that runs the process that listen to kafka to keeps the replica in sync.

Details

Provenance
douarddaAuthored on Apr 8 2019, 11:56 AM
douarddaPushed on Nov 26 2019, 1:34 PM
Differential Revision
D1393: Add support for a kafka-powered replica of the graph.
Parents
rDENV587f9df57a80: lister: Use explicit postgresql URLs as a workaround for a sqlalchemy issue
Branches
Unknown
Tags
Unknown