Page MenuHomeSoftware Heritage

Deploy a small publicly available kafka server (with some content) on a staging (+ the related objstorage)
Open, HighPublic


The idea is that I'd like to be able to document the mirror stack so that it "just works" out of the box so people interested in setting up a mirror can test it.

For this, we need a small kafka server with some data in it (say one or 2 small gitab/gitea ingested instances) that is publicly available. Only anonymized topics should be accessible there.

For the content-replayer to work, it will also need an objectstorage, since the content replayer pulls blobs from one objstorage to another (reading sha1s from the content topic).
This might be a bit trickier if we want to make it "read-only" to prevent kiddies from playing with it...
[edit] the ReadObjStorageFilter is our friend here[/edit]

It might also be useful to have the kafka accessible with and without authentication (to test that authentication layer), but it's not that important.

Event Timeline

douardda triaged this task as High priority.Fri, Oct 9, 3:37 PM
douardda created this task.
douardda updated the task description. (Show Details)Fri, Oct 9, 3:47 PM