Page MenuHomeSoftware Heritage
Differential D1345

swh.journal: Add backfiller implementation
ClosedPublic
Actions

Authored by ardumont on Apr 4 2019, 4:17 PM.

Details

Reviewers
olasd
vlorentz
douardda
Group Reviewers
Reviewers
Commits
rDJNL2f66a39e353c: Remove publisher notion
rDJNL469090993fea: backfill: Make partition key a singleton
rDJNL213c55cd856f: backfill: Flush the messages to the journal when done with the batch
rDJNL1609d9fffae7: backfill: Fix origin_visit support to use the right format
rDJNL5d1909ab16f4: backfill: Add test on parse_arguments
rDJNL9b46f5b9d225: backfill: Add test on origin_visit query computation
rDJNLfbcc6314eb10: backfill: Add test on skipped_content query computation
rDJNLdb38fcebd139: backfill: Add test on release query computation
rDJNLee359c5a203b: backfill: Add directory support
rDJNLa5fb0001c589: backfill: Use BaseDb instead of duplicating its code
rDJNL6a51cf41a385: backfill: Add snapshot support
rDJNL9a0c99fcc482: backfill: Add origin_visit support
rDJNLdf8842a8c1a2: backfill: Extract query computation into a function
rDJNLa7311e85bb11: backfill: Add release support
rDJNL5c2650dc8c4e: backfill: Add origin support
rDJNL603a9de09389: backfill: Add revision support
rDJNLe33661aae0f6: backfill: Support querying objects by type and range
rDJNLdc3202f1b55e: backfill: Add support for skipped_content
rDJNLf70956bbc629: backfill: Simplify constant names
rDJNL0e14c09b4cc1: backfill: Change implementation to send full object to journal
rDJNL99b238de4c04: backfill: Change initialization to explicit & check config
rDJNLb4509c44d719: backfill: Let the data flow
rDJNLc0b5d3df5e55: swh.journal: Rename checker to backfill
Summary

Backfill for the following object types:

  • content
  • skipped_content
  • directory
  • release
  • revision
  • snaphost
  • origin
  • origin_visit

Support for:

  • sending ranges
  • starting back from a given boundary (depending on the object type, could be a hash or an id).

This is not complete yet (missing directory and snapshot

We'd like to make sure the journal replayer is able to reuse the data sent to the journal.

Test Plan
  • tox
  • run in docker environment (D1346)

Diff Detail

Repository
rDJNL Journal infrastructure
Branch
backfiller
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 5218
Build 7053: tox-on-jenkinsJenkins
Build 7052: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
swh-public-ci added a comment.Apr 4 2019, 4:20 PM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/92/ for more details.

Harbormaster completed remote builds in B5103: Diff 4301.Apr 4 2019, 4:20 PM
ardumont planned changes to this revision.Apr 5 2019, 5:25 PM
ardumont updated this revision to Diff 4353.Apr 5 2019, 6:05 PM

Add support for origin, origin_visit, content, revision, release

(with pagination)

Remains snaphost, directory support to add

And checks with the KafkaReplayer

swh-public-ci added a comment.Apr 5 2019, 6:06 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/117/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/117/console

Harbormaster failed remote builds in B5156: Diff 4353!Apr 5 2019, 6:06 PM
ardumont mentioned this in D1346: docker-compose: Add journal-backfiller.Apr 5 2019, 6:06 PM
ardumont planned changes to this revision.Apr 5 2019, 11:54 PM
ardumont edited the summary of this revision. (Show Details)Apr 6 2019, 12:01 AM
ardumont edited the test plan for this revision. (Show Details)
ardumont added projects: Journal, Storage manager.
ardumont updated this revision to Diff 4387.Apr 8 2019, 3:12 PM
ardumont edited the test plan for this revision. (Show Details)

Add support for snapshot and directory

ardumont edited the summary of this revision. (Show Details)Apr 8 2019, 3:12 PM
swh-public-ci added a comment.Apr 8 2019, 3:12 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/121/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/121/console

Harbormaster failed remote builds in B5193: Diff 4387!Apr 8 2019, 3:12 PM
ardumont updated this revision to Diff 4390.Apr 8 2019, 3:16 PM

Fix pep8 violation

Apparently 100_000_000 is not supported in python3.5 but it is in latest one

swh-public-ci added a comment.Apr 8 2019, 3:19 PM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/122/ for more details.

Harbormaster completed remote builds in B5197: Diff 4390.Apr 8 2019, 3:19 PM
ardumont updated this revision to Diff 4393.Apr 8 2019, 3:52 PM
  • backfill: Fix origin_visit support to use the right format
  • backfill: Flush the messages to the journal when done with the batch
ardumont updated this revision to Diff 4394.Apr 8 2019, 3:54 PM
  • backfill: Make partition key a singleton
swh-public-ci added a comment.Apr 8 2019, 3:55 PM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/123/ for more details.

Harbormaster completed remote builds in B5201: Diff 4393.Apr 8 2019, 3:55 PM
swh-public-ci added a comment.Apr 8 2019, 3:58 PM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/124/ for more details.

Harbormaster completed remote builds in B5202: Diff 4394.Apr 8 2019, 3:58 PM
ardumont updated this revision to Diff 4402.Apr 9 2019, 11:54 AM

Rebase on latest master

swh-public-ci added a comment.Apr 9 2019, 11:57 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/126/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/126/console

Harbormaster failed remote builds in B5210: Diff 4402!Apr 9 2019, 11:57 AM
ardumont added a comment.Apr 9 2019, 12:03 PM

Build has FAILED

Yes, replayer tests are broken, unrelated to the backfiller
Apparently D1367 will fix that.

ardumont updated this revision to Diff 4409.Apr 9 2019, 12:19 PM

Rebase on cleanup_publisher branch

That gets out of the way some unrelated fix commits to the publisher

(tests will still fail)

ardumont updated this revision to Diff 4411.Apr 9 2019, 12:22 PM

Rebase on master

That gets out of the way the cli's logging setup fix (which is
unrelated to this branch and needed by other people anyway)

swh-public-ci added a comment.Apr 9 2019, 12:22 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/128/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/128/console

Harbormaster failed remote builds in B5215: Diff 4409!Apr 9 2019, 12:22 PM
ardumont updated this revision to Diff 4412.Apr 9 2019, 12:22 PM

oops, the rebase should have been on cleanup-publisher branch

done now

swh-public-ci added a comment.Apr 9 2019, 12:27 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/130/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/130/console

Harbormaster failed remote builds in B5218: Diff 4411!Apr 9 2019, 12:27 PM
swh-public-ci added a comment.Apr 9 2019, 12:30 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/131/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/131/console

Harbormaster failed remote builds in B5219: Diff 4412!Apr 9 2019, 12:30 PM
ardumont updated this revision to Diff 4430.Apr 9 2019, 4:22 PM
  • Remove publisher notion
swh-public-ci added a comment.Apr 9 2019, 4:26 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/133/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/133/console

Harbormaster failed remote builds in B5238: Diff 4430!Apr 9 2019, 4:26 PM
anlambert added a subscriber: anlambert.Apr 9 2019, 4:48 PM
anlambert added inline comments.
swh/journal/backfill.py
8

I do not really understand what the journal backfiller is supposed to do by reading that first sentence.

49

I think it should simply be "id" here. The defined alias is the same as the column name.

74

same here

78

same here

215–217

This could be moved in a dedicated _compute_shift_bits function as it is also used in the byte_ranges method below.

423

s/Reads/Read as imperative form is used elsewhere

ardumont added inline comments.Apr 9 2019, 5:46 PM
swh/journal/backfill.py
49

We used those to disambiguate the sql query we generate (when a join is implied).

For example, release, revision joins on person which also has columns name and id.

So we need those.

215–217

Indeed!
I will adapt.

ardumont updated this revision to Diff 4474.Apr 9 2019, 5:59 PM
  • backfill: Improve module docstrings
  • backfill: Flush when data is sent
  • backfill: Define a _compute_shit_bits function to reuse code
Harbormaster failed remote builds in B5282: Diff 4474!Apr 9 2019, 6:00 PM
swh-public-ci added a comment.Apr 9 2019, 6:00 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/136/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/136/console

ardumont updated this revision to Diff 4475.Apr 9 2019, 6:01 PM
ardumont marked an inline comment as done.

Rebase on latest master

swh-public-ci added a comment.Apr 9 2019, 6:01 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/137/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/137/console

Harbormaster failed remote builds in B5283: Diff 4475!Apr 9 2019, 6:01 PM
anlambert added inline comments.Apr 9 2019, 6:02 PM
swh/journal/backfill.py
49

Oh I see, that was not straightforward to understand.

ardumont updated this revision to Diff 4477.Apr 9 2019, 6:04 PM
  • backfill: Use the right logging instruction
  • backfill: Fix logging statement
ardumont added inline comments.Apr 9 2019, 6:07 PM
swh/journal/backfill.py
49

Indeed!

swh-public-ci added a comment.Apr 9 2019, 6:07 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/138/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/138/console

Harbormaster failed remote builds in B5285: Diff 4477!Apr 9 2019, 6:07 PM
ardumont added a subscriber: faux.Apr 11 2019, 10:00 AM
ardumont added inline comments.
swh/journal/backfill.py
340

@faux here :)

ardumont mentioned this in D1288: Storage.content_find returns list instead of single value.Apr 11 2019, 10:01 AM
ardumont updated this revision to Diff 4537.Apr 11 2019, 6:26 PM

Rebase on latest master

Harbormaster failed remote builds in B5345: Diff 4537!Apr 11 2019, 6:27 PM
swh-public-ci added a comment.Apr 11 2019, 6:27 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/141/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/141/console

ardumont updated this revision to Diff 4538.Apr 11 2019, 6:29 PM

Fix pep8 violation (missing one blank line)

Harbormaster completed remote builds in B5346: Diff 4538.Apr 11 2019, 6:33 PM
swh-public-ci added a comment.Apr 11 2019, 6:33 PM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/144/ for more details.

olasd accepted this revision.Apr 12 2019, 2:08 PM
This revision is now accepted and ready to land.Apr 12 2019, 2:08 PM
ardumont updated this revision to Diff 4546.Apr 12 2019, 2:10 PM

Rebase to latest master and branch diff to master

swh-public-ci added a comment.Apr 12 2019, 2:14 PM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/145/ for more details.

Harbormaster completed remote builds in B5375: Diff 4546.Apr 12 2019, 2:14 PM
Closed by commit rDJNLc0b5d3df5e55: swh.journal: Rename checker to backfill (authored by ardumont). · Explain WhyApr 12 2019, 2:21 PM
This revision was automatically updated to reflect the committed changes.
ardumont mentioned this in T1603: kafka storage backfiller.Apr 12 2019, 3:50 PM
ardumont retitled this revision from swh.journal: Bootstrap backfiller to swh.journal: Add backfiller implementation.

Revision Contents
Changeset List

Diff 4411

View Options

swh/journal/backfill.py

Loading...
View Options

swh/journal/checker.py

Loading...
View Options

swh/journal/cli.py

Loading...
View Options

swh/journal/direct_writer.py

Loading...
View Options

swh/journal/publisher.py

Loading...
View Options

swh/journal/tests/conftest.py

Loading...
View Options

swh/journal/tests/test_backfill.py

Loading...
View Options

swh/journal/tests/test_direct_writer.py

Loading...
View Options

swh/journal/tests/test_publisher_kafka.py

Loading...
View Options

swh/journal/tests/test_publisher_no_kafka.py

Loading...
About · Developer information · Legal