Paths

Table of Contentst

Differential D914

Notify Kafka when an origin visit is update.
ClosedPublic
Actions

Authored by vlorentz on Jan 10 2019, 4:26 PM.

Tags

None

Subscribers

Details

Reviewers

Group Reviewers

Commits

rDSTOCd19392c0fadb: Notify Kafka when an origin visit is update.
R65:d19392c0fadb: Notify Kafka when an origin visit is update.
rDSTOd19392c0fadb: Notify Kafka when an origin visit is update.

Summary

First step for T1465.

Note that it changes the value of topic_prefix.

Test Plan

There is a new test

Diff Detail

Repository

rDSTO Storage manager

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Jan 10 2019, 4:26 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJan 10 2019, 4:26 PM

vlorentz edited the summary of this revision. (Show Details)Jan 10 2019, 4:26 PM

vlorentz edited the test plan for this revision. (Show Details)

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/149/ for more details.

Harbormaster completed remote builds in B3388: Diff 2908.Jan 10 2019, 4:29 PM

vlorentz mentioned this in D916: Add support for object updates to the journal publisher..Jan 10 2019, 4:34 PM

I don't think there's any need to create the changed topics, or to add the mutable objects config : while its name doesn't reflect that, the new topic has always been designed as a way to notify the producer component of objects that need to be updated in the topic the actual clients subscribe to.

That's also apparent in D916 where the changed topic just ends up being used for feeding the same "final" topic.

To drive the point further : the journal hasn't been designed as a feed of all changes to the archive; it's currently designed as a way to get *the latest version* of all the objects that have been added to the archive, in a somewhat streaming fashion.

In the topics that the consumers are listening to, Kafka compaction rules will make messages about "old" versions of objects disappear when a new version (with an identical key) appears.

Drop the 'changed' topic.

olasd accepted this revision.Jan 10 2019, 4:50 PM

olasd added inline comments.

swh/storage/listener.py
69	maybe `"Registered to events for object type %s"` ?

This revision is now accepted and ready to land.Jan 10 2019, 4:50 PM

Improve log message

rebase + squash

Closed by commit rDSTOd19392c0fadb: Notify Kafka when an origin visit is update. (authored by vlorentz). · Explain WhyJan 10 2019, 4:51 PM

This revision was automatically updated to reflect the committed changes.

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/153/ for more details.

Harbormaster completed remote builds in B3392: Diff 2912.Jan 10 2019, 4:52 PM

Harbormaster failed remote builds in B3393: Diff 2913!

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/154/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/154/console

Harbormaster failed remote builds in B3394: Diff 2914!Jan 10 2019, 4:52 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/155/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/155/console

Revision Contents
Changeset List

Path

Size

swh/

storage/

15 lines

sql/

70-swh-triggers.sql

20 lines

tests/

test_listener.py

62 lines

Diff 2915

swh/storage/listener.py

Loading...

swh/storage/sql/70-swh-triggers.sql

Loading...

swh/storage/tests/test_listener.py

Loading...