Page MenuHomeSoftware Heritage
Differential D7360

Initialize DB schema and postgresql storage checker
ClosedPublic
Actions

Authored by vlorentz on Mar 16 2022, 12:34 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T3841: regularly scrub all the data stores of swh
Commits
rDSCRUBbe9a35c0c397: Initialize DB schema and postgresql storage checker
Test Plan

Will fail because they depend on D7359.

Diff Detail

Repository
rDSCRUB Datastore Scrubber
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Mar 16 2022, 12:34 PM
vlorentz published this revision for review.Mar 16 2022, 12:34 PM
swh-public-ci added a comment.Mar 16 2022, 12:34 PM

Build has FAILED

Patch application report for D7360 (id=26591)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 2f81a6f554357a040c539619fb7fdc004278de32
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/1/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/1/console

Harbormaster failed remote builds in B27471: Diff 26591!Mar 16 2022, 12:34 PM
vlorentz updated this revision to Diff 26592.Mar 16 2022, 1:31 PM

remove hypothesis, add missing deps

swh-public-ci added a comment.Mar 16 2022, 1:32 PM

Build has FAILED

Patch application report for D7360 (id=26592)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 356a9957843aa290d0acaea2255488c9419e1887
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/2/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/2/console

Harbormaster failed remote builds in B27472: Diff 26592!Mar 16 2022, 1:32 PM
vlorentz updated this revision to Diff 26593.Mar 16 2022, 1:37 PM

replace unnecessary test data

swh-public-ci added a comment.Mar 16 2022, 1:37 PM

Build has FAILED

Patch application report for D7360 (id=26593)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 5e1dce96bf29bb372f3bb677fbc77c99bb88d816
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/3/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/3/console

Harbormaster failed remote builds in B27473: Diff 26593!Mar 16 2022, 1:37 PM
vlorentz added a reviewer: Reviewers.Mar 16 2022, 2:28 PM
vlorentz updated this revision to Diff 26605.Mar 16 2022, 2:29 PM

add CLI test

vlorentz updated this revision to Diff 26606.Mar 16 2022, 2:29 PM

add missing file

swh-public-ci added a comment.Mar 16 2022, 2:29 PM

Build has FAILED

Patch application report for D7360 (id=26605)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit f982347f75d0d96bafc6c92f4dea6ca6a4dec737
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/4/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/4/console

Harbormaster failed remote builds in B27486: Diff 26605!Mar 16 2022, 2:29 PM
swh-public-ci added a comment.Mar 16 2022, 2:29 PM

Build has FAILED

Patch application report for D7360 (id=26606)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit ccd098c4436b7b09ccde5bd26363d5ff66344565
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/5/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/5/console

Harbormaster failed remote builds in B27487: Diff 26606!Mar 16 2022, 2:30 PM
vlorentz updated this revision to Diff 26608.Mar 16 2022, 2:55 PM

add test with more than 1 corrupt object

swh-public-ci added a comment.Mar 16 2022, 2:56 PM

Build has FAILED

Patch application report for D7360 (id=26608)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 785ee4f78ad45b87da0ace58b487a9046a990ed8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/6/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/6/console

Harbormaster failed remote builds in B27489: Diff 26608!Mar 16 2022, 2:56 PM
ardumont added a subscriber: ardumont.EditedMar 16 2022, 4:07 PM

Ok, i started reviewing this, it's long.
It sounds ok from afar, i stopped at the test_cli so far.

Can you please:

  • land the dependent this diff depends upon (afaict they are accepted now so after you can trigger again the build here).
  • link this diff to the related task

TIA

vlorentz added a task: T3841: regularly scrub all the data stores of swh.Mar 16 2022, 4:08 PM
vlorentz added a comment.EditedMar 16 2022, 4:11 PM

waiting for jenkins to release swh-storage

swh-public-ci added a comment.Mar 16 2022, 6:59 PM

Build has FAILED

Patch application report for D7360 (id=26608)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 785ee4f78ad45b87da0ace58b487a9046a990ed8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/7/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/7/console

swh-public-ci added a comment.Mar 17 2022, 1:27 PM

Build has FAILED

Patch application report for D7360 (id=26608)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 785ee4f78ad45b87da0ace58b487a9046a990ed8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/8/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/8/console

swh-public-ci added a comment.Mar 17 2022, 1:40 PM

Build has FAILED

Patch application report for D7360 (id=26608)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 785ee4f78ad45b87da0ace58b487a9046a990ed8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/9/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/9/console

vlorentz updated this revision to Diff 26636.Mar 17 2022, 1:45 PM

wat

swh-public-ci added a comment.Mar 17 2022, 1:46 PM

Build has FAILED

Patch application report for D7360 (id=26636)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit a8d2533f341b08ed5cbaf93fe1243e4d52e282c3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/10/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/10/console

Harbormaster failed remote builds in B27519: Diff 26636!Mar 17 2022, 1:46 PM
vlorentz updated this revision to Diff 26637.Mar 17 2022, 1:57 PM

fix yaml req

swh-public-ci added a comment.Mar 17 2022, 1:58 PM

Build has FAILED

Patch application report for D7360 (id=26637)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 72bcf77907d3c481e38a228b8997ef8a000db076
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/11/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/11/console

Harbormaster failed remote builds in B27520: Diff 26637!Mar 17 2022, 1:58 PM
vlorentz updated this revision to Diff 26638.Mar 17 2022, 2:03 PM

add types-pyyaml

swh-public-ci added a comment.Mar 17 2022, 2:05 PM

Build is green

Patch application report for D7360 (id=26638)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit be9a35c0c3977cb2b2f4ad2e9378d9cb394611f5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/12/ for more details.

Harbormaster completed remote builds in B27521: Diff 26638.Mar 17 2022, 2:05 PM
vlorentz added a child revision: D7368: Add a journal checker.Mar 17 2022, 3:14 PM
ardumont accepted this revision.Mar 17 2022, 3:34 PM

lgtm

couple of questions.

corrupted feels more correct than 'corrupt' (even though a quick survey and it looks to be ok in english too...)

swh/scrubber/db.py
25

?

swh/scrubber/tests/test_storage_postgresql.py
25

neat, i did not realize we could do that ;)

78

not a big fan of the object_ name...

105

That's actually the same test one on a specific snapshot, another on multiple snapshots.
Is it necessary to keep the first one?

This revision is now accepted and ready to land.Mar 17 2022, 3:34 PM
vlorentz added inline comments.Mar 17 2022, 3:39 PM
swh/scrubber/tests/test_storage_postgresql.py
78

I couldn't think of a better name. obj could work too, but it is discouraged by PEP 8:

If your public attribute name collides with a reserved keyword, append a single trailing underscore to your attribute name. This is preferable to an abbreviation or corrupted spelling.

https://peps.python.org/pep-0008/#designing-for-inheritance

105

the first one checks the checker does not report all objects after the first corrupt one, for example.

Closed by commit rDSCRUBbe9a35c0c397: Initialize DB schema and postgresql storage checker (authored by vlorentz). · Explain WhyMar 22 2022, 11:20 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSCRUBbe9a35c0c397: Initialize DB schema and postgresql storage checker.

Revision Contents
Changeset List

PathSize
1 line
1 line
docs/
34 lines
3 lines
3 lines
swh/
scrubber/
23 lines
91 lines
86 lines
89 lines
sql/
1 line
28 lines
13 lines
tests/
28 lines
64 lines
105 lines

Diff 26757

View Options

README.rst

Loading...
View Options

conftest.py

Loading...
View Options

docs/README.rst

Loading...
View Options

requirements-swh.txt

Loading...
View Options

requirements-test.txt

Loading...
View Options

swh/scrubber/__init__.py

Loading...
View Options

swh/scrubber/bar.py

Loading...
View Options

swh/scrubber/check_storage.py

Loading...
View Options

swh/scrubber/cli.py

Loading...
View Options

swh/scrubber/db.py

Loading...
View Options

swh/scrubber/sql/20-enums.sql

Loading...
View Options

swh/scrubber/sql/30-schema.sql

Loading...
View Options

swh/scrubber/sql/60-indexes.sql

Loading...
View Options

swh/scrubber/tests/conftest.py

Loading...
View Options

swh/scrubber/tests/test_cli.py

Loading...
View Options

swh/scrubber/tests/test_nothing.py

Loading...
View Options

swh/scrubber/tests/test_storage_postgresql.py

Loading...
About · Developer information · Legal