Page MenuHomeSoftware Heritage

docker-compose: Add journal stack
ClosedPublic

Authored by ardumont on Wed, Dec 19, 8:42 PM.

Details

Summary

Add:

  • journal (through kafka/zookeeper instance)
  • swh-storage-listener
  • swh-journal-publisher
  • swh-journal-client (dedicated client for the occasion)

Scenario is:

  • sql triggers on newly inserted objects (swh db) events
  • listener picked up those events, published those on the journal on temporary topics
  • publisher who is subscribed to the listener's topics, picked up those events, reified those associated objects, published them to the journal on new public topics
  • client who is subscribed on the origin/origin-visit public topics picked those up and logged them

Depends on D859
Depends on D863

Related T1443

Test Plan
docker-compose up

Load a repository, we should see logs about swh-journal-client logging events received (origin, origin-visit).

2018-12-20 13:20:35,344 1 INFO Setting newly assigned partitions {TopicPartition(topic='swh.journal.objects.origin_visit', partition=0), TopicPartition(topic='swh.journal.objects.origin', partition=0)} for group swh.journal.client
2018-12-20 13:20:35,404 1 INFO client received the following messages: defaultdict(<class 'list'>, {'origin_visit': [{b'date': b'2018-12-20 13:03:46.600807+00:00', b'metadata': None, b'origin': 2, b'snapshot': None, b'status': b'ongoing', b'visit': 4}, {b'date': b'2018-12-20 13:03:46.600807+00:00', b'metadata': None, b'origin': 2, b'snapshot': b'|4\x99n(\x8d\x98\xae\xa4\xed\xb2\x9eg\xaeT\x1a\xb2\x8b\xac\xd8', b'status': b'full', b'visit': 4}]})
2018-12-20 13:20:49,681 1 INFO client received the following messages: defaultdict(<class 'list'>, {'origin_visit': [{b'date': b'2018-12-20 13:20:33.234323+00:00', b'metadata': None, b'origin': 2, b'snapshot': None, b'status': b'ongoing', b'visit': 5}, {b'date': b'2018-12-20 13:20:33.234323+00:00', b'metadata': None, b'origin': 2, b'snapshot': b'|4\x99n(\x8d\x98\xae\xa4\xed\xb2\x9eg\xaeT\x1a\xb2\x8b\xac\xd8', b'status': b'full', b'visit': 5}]})

Diff Detail

Repository
rCDFD Dockerfiles for developers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Wed, Dec 19, 8:42 PM
ardumont planned changes to this revision.Wed, Dec 19, 8:42 PM
ardumont edited the summary of this revision. (Show Details)
ardumont updated this revision to Diff 2744.Thu, Dec 20, 12:47 AM
  • journal/listener: Fix default configuration
  • kafka.env: Simplify default configuration
  • swh-journal-publisher: Update default configuration
  • swh-journal-publisher: Run in debug mode
  • swh-journal-publisher: Simplify default configuration
ardumont planned changes to this revision.Thu, Dec 20, 12:47 AM
vlorentz accepted this revision.Thu, Dec 20, 11:23 AM
vlorentz added inline comments.
docker-compose.yml
21

why?

dockerfiles/swh-storage-listener/entrypoint.sh
30 ↗(On Diff #2744)

2>&1 to redirect stderr to stdout.

ardumont marked 2 inline comments as done.Thu, Dec 20, 11:31 AM
ardumont added inline comments.
docker-compose.yml
21

i don't know, i'm planning on checking if it's useful or not.

dockerfiles/swh-storage-listener/entrypoint.sh
30 ↗(On Diff #2744)

lol
Right you are!
thanks

It's not completely ready, i have for example:

  • the client that does nothing for the moment.
  • The listener and publisher are working but not completely.
  • The publisher somehow seems to not do anything in regards to origin/origin-visits
vlorentz added inline comments.Thu, Dec 20, 11:38 AM
docker-compose.yml
21

It's useful only if Kafka needs to spawn other containers. We don't do that, do we?

ardumont marked an inline comment as done.Thu, Dec 20, 12:16 PM
ardumont added inline comments.
docker-compose.yml
21

No, well not intentionally at least ;)

ardumont updated this revision to Diff 2752.Thu, Dec 20, 2:04 PM
  • publisher: Fix default configuration
  • Ignore docker-compose.override.yml
  • docker-compose: Remove unused volume
  • publisher: Fix default configuration, directory does not exist yet
  • swh-journal-client: Fix logging and configuration
This revision is now accepted and ready to land.Thu, Dec 20, 2:04 PM
ardumont edited the summary of this revision. (Show Details)Thu, Dec 20, 2:22 PM
ardumont edited the test plan for this revision. (Show Details)
ardumont updated this revision to Diff 2757.Thu, Dec 20, 2:31 PM
ardumont edited the test plan for this revision. (Show Details)
  • storage-listener: Fix .pgpass rights
ardumont edited the summary of this revision. (Show Details)Thu, Dec 20, 2:47 PM
ardumont edited the test plan for this revision. (Show Details)
This revision was automatically updated to reflect the committed changes.