Page MenuHomeSoftware Heritage

Notify Kafka when an origin visit is update.

Authored by vlorentz on Jan 10 2019, 4:26 PM.


Diff Detail

rDSTO Storage manager
Automatic diff as part of commit; lint not applicable.
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Jan 10 2019, 4:26 PM
vlorentz edited the summary of this revision. (Show Details)Jan 10 2019, 4:26 PM
vlorentz edited the test plan for this revision. (Show Details)
olasd added a subscriber: olasd.Jan 10 2019, 4:41 PM

I don't think there's any need to create the changed topics, or to add the mutable objects config : while its name doesn't reflect that, the new topic has always been designed as a way to notify the producer component of objects that need to be updated in the topic the actual clients subscribe to.

That's also apparent in D916 where the changed topic just ends up being used for feeding the same "final" topic.

olasd added a comment.Jan 10 2019, 4:46 PM

To drive the point further : the journal hasn't been designed as a feed of all changes to the archive; it's currently designed as a way to get *the latest version* of all the objects that have been added to the archive, in a somewhat streaming fashion.

In the topics that the consumers are listening to, Kafka compaction rules will make messages about "old" versions of objects disappear when a new version (with an identical key) appears.

vlorentz updated this revision to Diff 2912.Jan 10 2019, 4:48 PM
  • Drop the 'changed' topic.
olasd accepted this revision.Jan 10 2019, 4:50 PM
olasd added inline comments.

maybe "Registered to events for object type %s" ?

This revision is now accepted and ready to land.Jan 10 2019, 4:50 PM
vlorentz updated this revision to Diff 2913.Jan 10 2019, 4:51 PM
  • Improve log message
vlorentz updated this revision to Diff 2914.Jan 10 2019, 4:51 PM
  • rebase + squash
This revision was automatically updated to reflect the committed changes.
Harbormaster failed remote builds in B3393: Diff 2913!