Page MenuHomeSoftware Heritage

Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
ClosedPublic

Authored by vlorentz on Mar 18 2022, 3:38 PM.

Details

Summary

as well as the run() method.

For consistency with a future module called fixer.py

Diff Detail

Repository
rDSCRUB Datastore Scrubber
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 27622
Build 43233: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 43232: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D7391 (id=26699)

Could not rebase; Attempt merge onto 5e030a65a9...

Updating 5e030a6..33df97f
Fast-forward
 README.rst                                    |   1 +
 conftest.py                                   |   1 +
 docs/README.rst                               |  34 +++++++
 requirements-swh.txt                          |   3 +
 requirements-test.txt                         |   3 +
 swh/scrubber/__init__.py                      |  23 +++++
 swh/scrubber/bar.py                           |   4 -
 swh/scrubber/cli.py                           | 137 +++++++++++++++++++++++++-
 swh/scrubber/db.py                            |  95 ++++++++++++++++++
 swh/scrubber/journal_checker.py               |  69 +++++++++++++
 swh/scrubber/sql/20-enums.sql                 |   1 +
 swh/scrubber/sql/30-schema.sql                |  28 ++++++
 swh/scrubber/sql/60-indexes.sql               |  13 +++
 swh/scrubber/storage_checker.py               | 102 +++++++++++++++++++
 swh/scrubber/tests/conftest.py                |  28 ++++++
 swh/scrubber/tests/test_cli.py                | 115 +++++++++++++++++++++
 swh/scrubber/tests/test_journal_kafka.py      | 120 ++++++++++++++++++++++
 swh/scrubber/tests/test_nothing.py            |   3 -
 swh/scrubber/tests/test_storage_postgresql.py | 105 ++++++++++++++++++++
 19 files changed, 876 insertions(+), 9 deletions(-)
 create mode 120000 README.rst
 create mode 100644 conftest.py
 delete mode 100644 swh/scrubber/bar.py
 create mode 100644 swh/scrubber/db.py
 create mode 100644 swh/scrubber/journal_checker.py
 create mode 100644 swh/scrubber/sql/20-enums.sql
 create mode 100644 swh/scrubber/sql/30-schema.sql
 create mode 100644 swh/scrubber/sql/60-indexes.sql
 create mode 100644 swh/scrubber/storage_checker.py
 create mode 100644 swh/scrubber/tests/conftest.py
 create mode 100644 swh/scrubber/tests/test_cli.py
 create mode 100644 swh/scrubber/tests/test_journal_kafka.py
 delete mode 100644 swh/scrubber/tests/test_nothing.py
 create mode 100644 swh/scrubber/tests/test_storage_postgresql.py
Changes applied before test
commit 33df97f46afc886184a166895d8be1b4a1f72db7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

commit 30bef11d276ce509cf8fcee0489a5810690983ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Mar 17 14:48:49 2022 +0100

    Add a journal checker

commit be9a35c0c3977cb2b2f4ad2e9378d9cb394611f5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/14/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/14/console

Harbormaster returned this revision to the author for changes because remote builds failed.Mar 18 2022, 3:40 PM
Harbormaster failed remote builds in B27580: Diff 26699!
vlorentz retitled this revision from Rename check_{journal,storage}.py to {journal,storage}_checker.py to Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py.Mar 18 2022, 3:47 PM

Build has FAILED

Patch application report for D7391 (id=26701)

Could not rebase; Attempt merge onto 5e030a65a9...

Updating 5e030a6..20b7971
Fast-forward
 README.rst                                    |   1 +
 conftest.py                                   |   1 +
 docs/README.rst                               |  34 +++++++
 requirements-swh.txt                          |   3 +
 requirements-test.txt                         |   3 +
 swh/scrubber/__init__.py                      |  23 +++++
 swh/scrubber/bar.py                           |   4 -
 swh/scrubber/cli.py                           | 137 +++++++++++++++++++++++++-
 swh/scrubber/db.py                            |  96 ++++++++++++++++++
 swh/scrubber/journal_checker.py               |  69 +++++++++++++
 swh/scrubber/sql/20-enums.sql                 |   1 +
 swh/scrubber/sql/30-schema.sql                |  28 ++++++
 swh/scrubber/sql/60-indexes.sql               |  13 +++
 swh/scrubber/storage_checker.py               | 102 +++++++++++++++++++
 swh/scrubber/tests/conftest.py                |  28 ++++++
 swh/scrubber/tests/test_cli.py                | 115 +++++++++++++++++++++
 swh/scrubber/tests/test_journal_kafka.py      | 120 ++++++++++++++++++++++
 swh/scrubber/tests/test_nothing.py            |   3 -
 swh/scrubber/tests/test_storage_postgresql.py | 105 ++++++++++++++++++++
 19 files changed, 877 insertions(+), 9 deletions(-)
 create mode 120000 README.rst
 create mode 100644 conftest.py
 delete mode 100644 swh/scrubber/bar.py
 create mode 100644 swh/scrubber/db.py
 create mode 100644 swh/scrubber/journal_checker.py
 create mode 100644 swh/scrubber/sql/20-enums.sql
 create mode 100644 swh/scrubber/sql/30-schema.sql
 create mode 100644 swh/scrubber/sql/60-indexes.sql
 create mode 100644 swh/scrubber/storage_checker.py
 create mode 100644 swh/scrubber/tests/conftest.py
 create mode 100644 swh/scrubber/tests/test_cli.py
 create mode 100644 swh/scrubber/tests/test_journal_kafka.py
 delete mode 100644 swh/scrubber/tests/test_nothing.py
 create mode 100644 swh/scrubber/tests/test_storage_postgresql.py
Changes applied before test
commit 20b7971c8cc16b9c7e748d26e23d09bb7fc3658c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

commit 30bef11d276ce509cf8fcee0489a5810690983ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Mar 17 14:48:49 2022 +0100

    Add a journal checker

commit be9a35c0c3977cb2b2f4ad2e9378d9cb394611f5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/15/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/15/console

Harbormaster returned this revision to the author for changes because remote builds failed.Mar 18 2022, 3:49 PM
Harbormaster failed remote builds in B27597: Diff 26701!

Build has FAILED

Patch application report for D7391 (id=26701)

Could not rebase; Attempt merge onto 5e030a65a9...

Updating 5e030a6..20b7971
Fast-forward
 README.rst                                    |   1 +
 conftest.py                                   |   1 +
 docs/README.rst                               |  34 +++++++
 requirements-swh.txt                          |   3 +
 requirements-test.txt                         |   3 +
 swh/scrubber/__init__.py                      |  23 +++++
 swh/scrubber/bar.py                           |   4 -
 swh/scrubber/cli.py                           | 137 +++++++++++++++++++++++++-
 swh/scrubber/db.py                            |  96 ++++++++++++++++++
 swh/scrubber/journal_checker.py               |  69 +++++++++++++
 swh/scrubber/sql/20-enums.sql                 |   1 +
 swh/scrubber/sql/30-schema.sql                |  28 ++++++
 swh/scrubber/sql/60-indexes.sql               |  13 +++
 swh/scrubber/storage_checker.py               | 102 +++++++++++++++++++
 swh/scrubber/tests/conftest.py                |  28 ++++++
 swh/scrubber/tests/test_cli.py                | 115 +++++++++++++++++++++
 swh/scrubber/tests/test_journal_kafka.py      | 120 ++++++++++++++++++++++
 swh/scrubber/tests/test_nothing.py            |   3 -
 swh/scrubber/tests/test_storage_postgresql.py | 105 ++++++++++++++++++++
 19 files changed, 877 insertions(+), 9 deletions(-)
 create mode 120000 README.rst
 create mode 100644 conftest.py
 delete mode 100644 swh/scrubber/bar.py
 create mode 100644 swh/scrubber/db.py
 create mode 100644 swh/scrubber/journal_checker.py
 create mode 100644 swh/scrubber/sql/20-enums.sql
 create mode 100644 swh/scrubber/sql/30-schema.sql
 create mode 100644 swh/scrubber/sql/60-indexes.sql
 create mode 100644 swh/scrubber/storage_checker.py
 create mode 100644 swh/scrubber/tests/conftest.py
 create mode 100644 swh/scrubber/tests/test_cli.py
 create mode 100644 swh/scrubber/tests/test_journal_kafka.py
 delete mode 100644 swh/scrubber/tests/test_nothing.py
 create mode 100644 swh/scrubber/tests/test_storage_postgresql.py
Changes applied before test
commit 20b7971c8cc16b9c7e748d26e23d09bb7fc3658c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

commit 30bef11d276ce509cf8fcee0489a5810690983ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Mar 17 14:48:49 2022 +0100

    Add a journal checker

commit be9a35c0c3977cb2b2f4ad2e9378d9cb394611f5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/16/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/16/console

Build is green

Patch application report for D7391 (id=26727)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 9d75d5f0989b2ae54d1c1b13e3b77d46668f50b3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

commit 30bef11d276ce509cf8fcee0489a5810690983ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Mar 17 14:48:49 2022 +0100

    Add a journal checker

commit be9a35c0c3977cb2b2f4ad2e9378d9cb394611f5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/17/ for more details.

I recall having read most of that diff already, didn't you include too much in your diff?

I recall having read most of that diff already, didn't you include too much in your diff?

@vlorentz ping ^ (i don't want to read it all again if i can help it) Can you please check your commit range?

Build is green

Patch application report for D7391 (id=26810)

Rebasing onto f4214c2f7a...

First, rewinding head to replay your work on top of it...
Applying: Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
Changes applied before test
commit 99de7d6c22bddc9e61cd9556800a29222b4e36b0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/18/ for more details.

This revision is now accepted and ready to land.Mar 23 2022, 2:56 PM
This revision was landed with ongoing or failed builds.Mar 24 2022, 3:46 PM
This revision was automatically updated to reflect the committed changes.

Build is green

Patch application report for D7391 (id=26867)

Rebasing onto f4214c2f7a...

Current branch diff-target is up to date.
Changes applied before test
commit 4144ac789ff8a65380ce759ff375612e9a63c1f4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Mar 18 11:46:27 2022 +0100

    Add doc + rename check_{journal,storage}.py to {journal,storage}_checker.py
    
    as well as the run() method.
    
    For consistency with a future module called fixer.py

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/19/ for more details.