Page MenuHomeSoftware Heritage
Differential D7391

Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
ClosedPublic
Actions

Authored by vlorentz on Mar 18 2022, 3:38 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDSCRUB4144ac789ff8: Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
Summary

as well as the run() method.

For consistency with a future module called fixer.py

Diff Detail

Repository
rDSCRUB Datastore Scrubber
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 27622
Build 43233: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 43232: arc lint + arc unit

Event Timeline

vlorentz created this revision.Mar 18 2022, 3:38 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMar 18 2022, 3:38 PM
swh-public-ci added a comment.Mar 18 2022, 3:40 PM

Build has FAILED

Patch application report for D7391 (id=26699)

Could not rebase; Attempt merge onto 5e030a65a9...

Updating 5e030a6..33df97f
Fast-forward
 README.rst                                    |   1 +
 conftest.py                                   |   1 +
 docs/README.rst                               |  34 +++++++
 requirements-swh.txt                          |   3 +
 requirements-test.txt                         |   3 +
 swh/scrubber/__init__.py                      |  23 +++++
 swh/scrubber/bar.py                           |   4 -
 swh/scrubber/cli.py                           | 137 +++++++++++++++++++++++++-
 swh/scrubber/db.py                            |  95 ++++++++++++++++++
 swh/scrubber/journal_checker.py               |  69 +++++++++++++
 swh/scrubber/sql/20-enums.sql                 |   1 +
 swh/scrubber/sql/30-schema.sql                |  28 ++++++
 swh/scrubber/sql/60-indexes.sql               |  13 +++
 swh/scrubber/storage_checker.py               | 102 +++++++++++++++++++
 swh/scrubber/tests/conftest.py                |  28 ++++++
 swh/scrubber/tests/test_cli.py                | 115 +++++++++++++++++++++
 swh/scrubber/tests/test_journal_kafka.py      | 120 ++++++++++++++++++++++
 swh/scrubber/tests/test_nothing.py            |   3 -
 swh/scrubber/tests/test_storage_postgresql.py | 105 ++++++++++++++++++++
 19 files changed, 876 insertions(+), 9 deletions(-)
 create mode 120000 README.rst
 create mode 100644 conftest.py
 delete mode 100644 swh/scrubber/bar.py
 create mode 100644 swh/scrubber/db.py
 create mode 100644 swh/scrubber/journal_checker.py
 create mode 100644 swh/scrubber/sql/20-enums.sql
 create mode 100644 swh/scrubber/sql/30-schema.sql
 create mode 100644 swh/scrubber/sql/60-indexes.sql
 create mode 100644 swh/scrubber/storage_checker.py
 create mode 100644 swh/scrubber/tests/conftest.py
 create mode 100644 swh/scrubber/tests/test_cli.py
 create mode 100644 swh/scrubber/tests/test_journal_kafka.py
 delete mode 100644 swh/scrubber/tests/test_nothing.py
 create mode 100644 swh/scrubber/tests/test_storage_postgresql.py
Changes applied before test
commit 33df97f46afc886184a166895d8be1b4a1f72db7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

commit 30bef11d276ce509cf8fcee0489a5810690983ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Mar 17 14:48:49 2022 +0100

    Add a journal checker

commit be9a35c0c3977cb2b2f4ad2e9378d9cb394611f5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/14/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/14/console

Harbormaster returned this revision to the author for changes because remote builds failed.Mar 18 2022, 3:40 PM
Harbormaster failed remote builds in B27580: Diff 26699!
vlorentz updated this revision to Diff 26701.Mar 18 2022, 3:47 PM

fix build

vlorentz retitled this revision from Rename check_{journal,storage}.py to {journal,storage}_checker.py to Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py.Mar 18 2022, 3:47 PM
swh-public-ci added a comment.Mar 18 2022, 3:49 PM

Build has FAILED

Patch application report for D7391 (id=26701)

Could not rebase; Attempt merge onto 5e030a65a9...

Updating 5e030a6..20b7971
Fast-forward
 README.rst                                    |   1 +
 conftest.py                                   |   1 +
 docs/README.rst                               |  34 +++++++
 requirements-swh.txt                          |   3 +
 requirements-test.txt                         |   3 +
 swh/scrubber/__init__.py                      |  23 +++++
 swh/scrubber/bar.py                           |   4 -
 swh/scrubber/cli.py                           | 137 +++++++++++++++++++++++++-
 swh/scrubber/db.py                            |  96 ++++++++++++++++++
 swh/scrubber/journal_checker.py               |  69 +++++++++++++
 swh/scrubber/sql/20-enums.sql                 |   1 +
 swh/scrubber/sql/30-schema.sql                |  28 ++++++
 swh/scrubber/sql/60-indexes.sql               |  13 +++
 swh/scrubber/storage_checker.py               | 102 +++++++++++++++++++
 swh/scrubber/tests/conftest.py                |  28 ++++++
 swh/scrubber/tests/test_cli.py                | 115 +++++++++++++++++++++
 swh/scrubber/tests/test_journal_kafka.py      | 120 ++++++++++++++++++++++
 swh/scrubber/tests/test_nothing.py            |   3 -
 swh/scrubber/tests/test_storage_postgresql.py | 105 ++++++++++++++++++++
 19 files changed, 877 insertions(+), 9 deletions(-)
 create mode 120000 README.rst
 create mode 100644 conftest.py
 delete mode 100644 swh/scrubber/bar.py
 create mode 100644 swh/scrubber/db.py
 create mode 100644 swh/scrubber/journal_checker.py
 create mode 100644 swh/scrubber/sql/20-enums.sql
 create mode 100644 swh/scrubber/sql/30-schema.sql
 create mode 100644 swh/scrubber/sql/60-indexes.sql
 create mode 100644 swh/scrubber/storage_checker.py
 create mode 100644 swh/scrubber/tests/conftest.py
 create mode 100644 swh/scrubber/tests/test_cli.py
 create mode 100644 swh/scrubber/tests/test_journal_kafka.py
 delete mode 100644 swh/scrubber/tests/test_nothing.py
 create mode 100644 swh/scrubber/tests/test_storage_postgresql.py
Changes applied before test
commit 20b7971c8cc16b9c7e748d26e23d09bb7fc3658c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

commit 30bef11d276ce509cf8fcee0489a5810690983ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Mar 17 14:48:49 2022 +0100

    Add a journal checker

commit be9a35c0c3977cb2b2f4ad2e9378d9cb394611f5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/15/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/15/console

Harbormaster returned this revision to the author for changes because remote builds failed.Mar 18 2022, 3:49 PM
Harbormaster failed remote builds in B27597: Diff 26701!
swh-public-ci added a comment.Mar 18 2022, 3:55 PM

Build has FAILED

Patch application report for D7391 (id=26701)

Could not rebase; Attempt merge onto 5e030a65a9...

Updating 5e030a6..20b7971
Fast-forward
 README.rst                                    |   1 +
 conftest.py                                   |   1 +
 docs/README.rst                               |  34 +++++++
 requirements-swh.txt                          |   3 +
 requirements-test.txt                         |   3 +
 swh/scrubber/__init__.py                      |  23 +++++
 swh/scrubber/bar.py                           |   4 -
 swh/scrubber/cli.py                           | 137 +++++++++++++++++++++++++-
 swh/scrubber/db.py                            |  96 ++++++++++++++++++
 swh/scrubber/journal_checker.py               |  69 +++++++++++++
 swh/scrubber/sql/20-enums.sql                 |   1 +
 swh/scrubber/sql/30-schema.sql                |  28 ++++++
 swh/scrubber/sql/60-indexes.sql               |  13 +++
 swh/scrubber/storage_checker.py               | 102 +++++++++++++++++++
 swh/scrubber/tests/conftest.py                |  28 ++++++
 swh/scrubber/tests/test_cli.py                | 115 +++++++++++++++++++++
 swh/scrubber/tests/test_journal_kafka.py      | 120 ++++++++++++++++++++++
 swh/scrubber/tests/test_nothing.py            |   3 -
 swh/scrubber/tests/test_storage_postgresql.py | 105 ++++++++++++++++++++
 19 files changed, 877 insertions(+), 9 deletions(-)
 create mode 120000 README.rst
 create mode 100644 conftest.py
 delete mode 100644 swh/scrubber/bar.py
 create mode 100644 swh/scrubber/db.py
 create mode 100644 swh/scrubber/journal_checker.py
 create mode 100644 swh/scrubber/sql/20-enums.sql
 create mode 100644 swh/scrubber/sql/30-schema.sql
 create mode 100644 swh/scrubber/sql/60-indexes.sql
 create mode 100644 swh/scrubber/storage_checker.py
 create mode 100644 swh/scrubber/tests/conftest.py
 create mode 100644 swh/scrubber/tests/test_cli.py
 create mode 100644 swh/scrubber/tests/test_journal_kafka.py
 delete mode 100644 swh/scrubber/tests/test_nothing.py
 create mode 100644 swh/scrubber/tests/test_storage_postgresql.py
Changes applied before test
commit 20b7971c8cc16b9c7e748d26e23d09bb7fc3658c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

commit 30bef11d276ce509cf8fcee0489a5810690983ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Mar 17 14:48:49 2022 +0100

    Add a journal checker

commit be9a35c0c3977cb2b2f4ad2e9378d9cb394611f5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/16/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/16/console

vlorentz updated this revision to Diff 26727.Mar 21 2022, 9:47 AM

fix build again

swh-public-ci added a comment.Mar 21 2022, 9:49 AM

Build is green

Patch application report for D7391 (id=26727)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 9d75d5f0989b2ae54d1c1b13e3b77d46668f50b3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

commit 30bef11d276ce509cf8fcee0489a5810690983ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Mar 17 14:48:49 2022 +0100

    Add a journal checker

commit be9a35c0c3977cb2b2f4ad2e9378d9cb394611f5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/17/ for more details.

Harbormaster completed remote builds in B27622: Diff 26727.Mar 21 2022, 9:49 AM
vlorentz requested review of this revision.Mar 21 2022, 9:49 AM
ardumont added a subscriber: ardumont.Mar 21 2022, 9:55 AM

I recall having read most of that diff already, didn't you include too much in your diff?

ardumont added a comment.Mar 23 2022, 9:31 AM

I recall having read most of that diff already, didn't you include too much in your diff?

@vlorentz ping ^ (i don't want to read it all again if i can help it) Can you please check your commit range?

vlorentz updated this revision to Diff 26810.Mar 23 2022, 10:09 AM

remove old commits

swh-public-ci added a comment.Mar 23 2022, 10:11 AM

Build is green

Patch application report for D7391 (id=26810)

Rebasing onto f4214c2f7a...

First, rewinding head to replay your work on top of it...
Applying: Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
Changes applied before test
commit 99de7d6c22bddc9e61cd9556800a29222b4e36b0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/18/ for more details.

Harbormaster completed remote builds in B27730: Diff 26810.Mar 23 2022, 10:11 AM
ardumont accepted this revision.Mar 23 2022, 2:56 PM

Thanks.

This revision is now accepted and ready to land.Mar 23 2022, 2:56 PM
vlorentz updated this revision to Diff 26867.Mar 24 2022, 3:46 PM

rebase

This revision was landed with ongoing or failed builds.Mar 24 2022, 3:46 PM
Closed by commit rDSCRUB4144ac789ff8: Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSCRUB4144ac789ff8: Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py.
swh-public-ci added a comment.Mar 24 2022, 3:48 PM

Build is green

Patch application report for D7391 (id=26867)

Rebasing onto f4214c2f7a...

Current branch diff-target is up to date.
Changes applied before test
commit 4144ac789ff8a65380ce759ff375612e9a63c1f4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/19/ for more details.

Harbormaster completed remote builds in B27781: Diff 26867.Mar 24 2022, 3:48 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
1 line--
1 line--
docs/
34 lines--
3 lines--
3 lines--
swh/
scrubber/
23 lines50%
Loading...
--
137 lines87%
Loading...
96 lines100%
Loading...
69 lines96%
Loading...
sql/
1 line--
28 lines--
13 lines--
102 lines95%
Loading...
tests/
28 lines100%
Loading...
115 lines100%
Loading...
120 lines100%
Loading...
--
105 lines100%
Loading...

Diff 26727

View Options

README.rst

Loading...
View Options

conftest.py

Loading...
View Options

docs/README.rst

Loading...
View Options

requirements-swh.txt

Loading...
View Options

requirements-test.txt

Loading...
View Options

swh/scrubber/__init__.py

Loading...
View Options

swh/scrubber/bar.py

Loading...
View Options

swh/scrubber/cli.py

Loading...
View Options

swh/scrubber/db.py

Loading...
View Options

swh/scrubber/journal_checker.py

Loading...
View Options

swh/scrubber/sql/20-enums.sql

Loading...
View Options

swh/scrubber/sql/30-schema.sql

Loading...
View Options

swh/scrubber/sql/60-indexes.sql

Loading...
View Options

swh/scrubber/storage_checker.py

Loading...
View Options

swh/scrubber/tests/conftest.py

Loading...
View Options

swh/scrubber/tests/test_cli.py

Loading...
View Options

swh/scrubber/tests/test_journal_kafka.py

Loading...
View Options

swh/scrubber/tests/test_nothing.py

Loading...
View Options

swh/scrubber/tests/test_storage_postgresql.py

Loading...
About · Developer information · Legal