Page MenuHomeSoftware Heritage

Notify Kafka when an origin visit is update.
ClosedPublic

Authored by vlorentz on Jan 10 2019, 4:26 PM.

Details

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz edited the test plan for this revision. (Show Details)

I don't think there's any need to create the changed topics, or to add the mutable objects config : while its name doesn't reflect that, the new topic has always been designed as a way to notify the producer component of objects that need to be updated in the topic the actual clients subscribe to.

That's also apparent in D916 where the changed topic just ends up being used for feeding the same "final" topic.

To drive the point further : the journal hasn't been designed as a feed of all changes to the archive; it's currently designed as a way to get *the latest version* of all the objects that have been added to the archive, in a somewhat streaming fashion.

In the topics that the consumers are listening to, Kafka compaction rules will make messages about "old" versions of objects disappear when a new version (with an identical key) appears.

  • Drop the 'changed' topic.
olasd added inline comments.
swh/storage/listener.py
69

maybe "Registered to events for object type %s" ?

This revision is now accepted and ready to land.Jan 10 2019, 4:50 PM
This revision was automatically updated to reflect the committed changes.