Page MenuHomeSoftware Heritage
Differential D7589

Add a --margin option to the `swh dataset graph export` command
ClosedPublic
Actions

Authored by douardda on Apr 15 2022, 12:29 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDDATASET07bcf1674e0a: Add a --margin option to the `swh dataset graph export` command
Summary

this option allows to restart consuming kafka a bit earlier than last
committed offsets.
This can be useful to test and debug.

Depends on D7588.

Diff Detail

Event Timeline

douardda created this revision.Apr 15 2022, 12:29 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 15 2022, 12:29 PM
swh-public-ci added a comment.Apr 15 2022, 12:31 PM

Build is green

Patch application report for D7589 (id=27476)

Could not rebase; Attempt merge onto 9f342d9994...

Updating 9f342d9..6d07ab9
Fast-forward
 swh/dataset/cli.py              | 27 +++++++++++++++++++++++++--
 swh/dataset/journalprocessor.py | 21 ++++++++++++++++++---
 2 files changed, 43 insertions(+), 5 deletions(-)
Changes applied before test
commit 6d07ab9fc745bd8375f4ece73459fdadc87f7f28
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 15 12:25:47 2022 +0200

    Add a --margin option to the `swh dataset graph export` command
    
    this option allows to restart consuming kafka a bit earlier than last
    committed offsets.
    This can be useful to test and debug.

commit 6f6b1bd42760203427d2b8f72a6fd22d4686b106
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 15 12:19:17 2022 +0200

    Add a --type option to the `swh dataset graph export` command
    
    to easily set the list of exported object types. If not set, export all
    supported object types.
    
    Note that ``--exclude`` is also applied.

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/131/ for more details.

Harbormaster completed remote builds in B28494: Diff 27476.Apr 15 2022, 12:31 PM
douardda requested review of this revision.Apr 15 2022, 12:31 PM
douardda edited the summary of this revision. (Show Details)Apr 15 2022, 12:54 PM
douardda added a parent revision: D7588: Add a --type option to the `swh dataset graph export` command.
douardda removed a parent revision: D5788: common/origin_save: Update missing information when available.
douardda updated this revision to Diff 27483.Apr 15 2022, 1:27 PM

Improve and simplify a bit the code

so that progress bars are not messed up.

swh-public-ci added a comment.Apr 15 2022, 1:29 PM

Build is green

Patch application report for D7589 (id=27483)

Could not rebase; Attempt merge onto 9f342d9994...

Updating 9f342d9..08eac9d
Fast-forward
 swh/dataset/cli.py              | 26 ++++++++++++++++++++++++--
 swh/dataset/journalprocessor.py | 27 ++++++++++++++++++++++++---
 2 files changed, 48 insertions(+), 5 deletions(-)
Changes applied before test
commit 08eac9da7cabae2c018edb00ac44b10351e12ef2
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 15 12:25:47 2022 +0200

    Add a --margin option to the `swh dataset graph export` command
    
    this option allows to restart consuming kafka a bit earlier than last
    committed offsets.
    This can be useful to test and debug.

commit 6f6b1bd42760203427d2b8f72a6fd22d4686b106
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 15 12:19:17 2022 +0200

    Add a --type option to the `swh dataset graph export` command
    
    to easily set the list of exported object types. If not set, export all
    supported object types.
    
    Note that ``--exclude`` is also applied.

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/132/ for more details.

Harbormaster completed remote builds in B28501: Diff 27483.Apr 15 2022, 1:29 PM
vlorentz added a subscriber: vlorentz.Apr 20 2022, 11:27 AM

Sounds like you're having fun

swh/dataset/cli.py
87

https://click.palletsprojects.com/en/8.0.x/options/#range-options

swh/dataset/journalprocessor.py
257

I don't think lo would ever be large enough to overflow a double mantissa, but better safe than sorry.

vlorentz accepted this revision.Apr 20 2022, 11:29 AM
This revision is now accepted and ready to land.Apr 20 2022, 11:29 AM
douardda marked 2 inline comments as done.Apr 21 2022, 9:44 AM
douardda added inline comments.
swh/dataset/cli.py
87

whuu nice!

douardda updated this revision to Diff 27563.Apr 21 2022, 9:46 AM
douardda marked an inline comment as done.

thx vlorentz

swh-public-ci added a comment.Apr 21 2022, 9:49 AM

Build is green

Patch application report for D7589 (id=27563)

Could not rebase; Attempt merge onto 9f342d9994...

Updating 9f342d9..628e561
Fast-forward
 swh/dataset/cli.py              | 26 ++++++++++++++++++++++++--
 swh/dataset/journalprocessor.py | 29 ++++++++++++++++++++++++++---
 2 files changed, 50 insertions(+), 5 deletions(-)
Changes applied before test
commit 628e561e778f9b328b5b1afc5c070f25f55e6d98
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 15 12:25:47 2022 +0200

    Add a --margin option to the `swh dataset graph export` command
    
    this option allows to restart consuming kafka a bit earlier than last
    committed offsets.
    This can be useful to test and debug.

commit 6f6b1bd42760203427d2b8f72a6fd22d4686b106
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 15 12:19:17 2022 +0200

    Add a --type option to the `swh dataset graph export` command
    
    to easily set the list of exported object types. If not set, export all
    supported object types.
    
    Note that ``--exclude`` is also applied.

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/133/ for more details.

Harbormaster completed remote builds in B28592: Diff 27563.Apr 21 2022, 9:49 AM
douardda updated this revision to Diff 27569.Apr 21 2022, 9:52 AM

rebase

swh-public-ci added a comment.Apr 21 2022, 9:54 AM

Build is green

Patch application report for D7589 (id=27569)

Could not rebase; Attempt merge onto 9f342d9994...

Updating 9f342d9..07bcf16
Fast-forward
 swh/dataset/cli.py              | 32 ++++++++++++++++++++++++++++++--
 swh/dataset/journalprocessor.py | 29 ++++++++++++++++++++++++++---
 2 files changed, 56 insertions(+), 5 deletions(-)
Changes applied before test
commit 07bcf1674e0a7873f44c6b71a0f7047c58f9d402
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 15 12:25:47 2022 +0200

    Add a --margin option to the `swh dataset graph export` command
    
    this option allows to restart consuming kafka a bit earlier than last
    committed offsets.
    This can be useful to test and debug.

commit 4154d43a4c55c20df7f37466f84adba3c1505c64
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 15 12:19:17 2022 +0200

    Add a --types option to the `swh dataset graph export` command
    
    to easily set the list of exported object types. If not set, export all
    supported object types.
    
    Note that ``--exclude`` is also applied.

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/135/ for more details.

Harbormaster completed remote builds in B28598: Diff 27569.Apr 21 2022, 9:54 AM
Closed by commit rDDATASET07bcf1674e0a: Add a --margin option to the `swh dataset graph export` command (authored by douardda). · Explain WhyApr 21 2022, 10:10 AM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDDATASET07bcf1674e0a: Add a --margin option to the `swh dataset graph export` command.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
dataset/
12 lines49%
Loading...
29 lines19%
Loading...

Diff 27569

View Options

swh/dataset/cli.py

Loading...
View Options

swh/dataset/journalprocessor.py

Loading...
About · Developer information · Legal