Page MenuHomeSoftware Heritage

Notify Kafka when an origin visit is update.
ClosedPublic

Authored by vlorentz on Thu, Jan 10, 4:26 PM.

Details

Summary

First step for T1465.

Note that it changes the value of topic_prefix.

Test Plan

There is a new test

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Thu, Jan 10, 4:26 PM
vlorentz edited the summary of this revision. (Show Details)Thu, Jan 10, 4:26 PM
vlorentz edited the test plan for this revision. (Show Details)
olasd added a subscriber: olasd.Thu, Jan 10, 4:41 PM

I don't think there's any need to create the changed topics, or to add the mutable objects config : while its name doesn't reflect that, the new topic has always been designed as a way to notify the producer component of objects that need to be updated in the topic the actual clients subscribe to.

That's also apparent in D916 where the changed topic just ends up being used for feeding the same "final" topic.

olasd added a comment.Thu, Jan 10, 4:46 PM

To drive the point further : the journal hasn't been designed as a feed of all changes to the archive; it's currently designed as a way to get *the latest version* of all the objects that have been added to the archive, in a somewhat streaming fashion.

In the topics that the consumers are listening to, Kafka compaction rules will make messages about "old" versions of objects disappear when a new version (with an identical key) appears.

vlorentz updated this revision to Diff 2912.Thu, Jan 10, 4:48 PM
  • Drop the 'changed' topic.
olasd accepted this revision.Thu, Jan 10, 4:50 PM
olasd added inline comments.
swh/storage/listener.py
70

maybe "Registered to events for object type %s" ?

This revision is now accepted and ready to land.Thu, Jan 10, 4:50 PM
vlorentz updated this revision to Diff 2913.Thu, Jan 10, 4:51 PM
  • Improve log message
vlorentz updated this revision to Diff 2914.Thu, Jan 10, 4:51 PM
  • rebase + squash
This revision was automatically updated to reflect the committed changes.
Harbormaster failed remote builds in B3393: Diff 2913!