Details

Reviewers

olasd
douardda
vlorentz

Group Reviewers

Reviewers

Commits

Summary

Add:

journal (through kafka/zookeeper instance)
swh-storage-listener
swh-journal-publisher
swh-journal-client (dedicated client for the occasion)

Scenario is:

sql triggers on newly inserted objects (swh db) events
listener picked up those events, published those on the journal on temporary topics
publisher who is subscribed to the listener's topics, picked up those events, reified those associated objects, published them to the journal on new public topics
client who is subscribed on the origin/origin-visit public topics picked those up and logged them

Depends on D859
Depends on D863

Related T1443

Test Plan

docker-compose up

Load a repository, we should see logs about swh-journal-client logging events received (origin, origin-visit).

2018-12-20 13:20:35,344 1 INFO Setting newly assigned partitions {TopicPartition(topic='swh.journal.objects.origin_visit', partition=0), TopicPartition(topic='swh.journal.objects.origin', partition=0)} for group swh.journal.client
2018-12-20 13:20:35,404 1 INFO client received the following messages: defaultdict(<class 'list'>, {'origin_visit': [{b'date': b'2018-12-20 13:03:46.600807+00:00', b'metadata': None, b'origin': 2, b'snapshot': None, b'status': b'ongoing', b'visit': 4}, {b'date': b'2018-12-20 13:03:46.600807+00:00', b'metadata': None, b'origin': 2, b'snapshot': b'|4\x99n(\x8d\x98\xae\xa4\xed\xb2\x9eg\xaeT\x1a\xb2\x8b\xac\xd8', b'status': b'full', b'visit': 4}]})
2018-12-20 13:20:49,681 1 INFO client received the following messages: defaultdict(<class 'list'>, {'origin_visit': [{b'date': b'2018-12-20 13:20:33.234323+00:00', b'metadata': None, b'origin': 2, b'snapshot': None, b'status': b'ongoing', b'visit': 5}, {b'date': b'2018-12-20 13:20:33.234323+00:00', b'metadata': None, b'origin': 2, b'snapshot': b'|4\x99n(\x8d\x98\xae\xa4\xed\xb2\x9eg\xaeT\x1a\xb2\x8b\xac\xd8', b'status': b'full', b'visit': 5}]})

Diff Detail

Repository

rCDFD Dockerfiles for developers

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Dec 19 2018, 8:42 PM

Harbormaster completed remote builds in B3185: Diff 2742.Dec 19 2018, 8:42 PM

ardumont planned changes to this revision.Dec 19 2018, 8:42 PM

ardumont edited the summary of this revision. (Show Details)

ardumont added a parent revision: D859: listener: fix decoding behavior and add logging behavior to the service.

journal/listener: Fix default configuration
kafka.env: Simplify default configuration
swh-journal-publisher: Update default configuration
swh-journal-publisher: Run in debug mode
swh-journal-publisher: Simplify default configuration

Harbormaster completed remote builds in B3187: Diff 2744.Dec 20 2018, 12:47 AM

ardumont planned changes to this revision.Dec 20 2018, 12:47 AM

vlorentz accepted this revision.Dec 20 2018, 11:23 AM

vlorentz added inline comments.

docker-compose.yml
20	why?
dockerfiles/swh-storage-listener/entrypoint.sh
30 ↗	(On Diff #2744)	`2>&1` to redirect stderr to stdout.

ardumont marked 2 inline comments as done.Dec 20 2018, 11:31 AM

ardumont added inline comments.

docker-compose.yml
20	i don't know, i'm planning on checking if it's useful or not.
dockerfiles/swh-storage-listener/entrypoint.sh
30 ↗	(On Diff #2744)	lol Right you are! thanks

It's not completely ready, i have for example:

the client that does nothing for the moment.
The listener and publisher are working but not completely.
The publisher somehow seems to not do anything in regards to origin/origin-visits

vlorentz added inline comments.Dec 20 2018, 11:38 AM