Page MenuHomeSoftware Heritage
Differential D7360

Initialize DB schema and postgresql storage checker
ClosedPublic
Actions

Authored by vlorentz on Mar 16 2022, 12:34 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T3841: regularly scrub all the data stores of swh
Commits
rDSCRUBbe9a35c0c397: Initialize DB schema and postgresql storage checker
Test Plan

Will fail because they depend on D7359.

Diff Detail

Repository
rDSCRUB Datastore Scrubber
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 27489
Build 43014: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 43013: arc lint + arc unit

Event Timeline

vlorentz created this revision.Mar 16 2022, 12:34 PM
vlorentz published this revision for review.Mar 16 2022, 12:34 PM
swh-public-ci added a comment.Mar 16 2022, 12:34 PM

Build has FAILED

Patch application report for D7360 (id=26591)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 2f81a6f554357a040c539619fb7fdc004278de32
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/1/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/1/console

Harbormaster failed remote builds in B27471: Diff 26591!Mar 16 2022, 12:34 PM
vlorentz updated this revision to Diff 26592.Mar 16 2022, 1:31 PM

remove hypothesis, add missing deps

swh-public-ci added a comment.Mar 16 2022, 1:32 PM

Build has FAILED

Patch application report for D7360 (id=26592)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 356a9957843aa290d0acaea2255488c9419e1887
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/2/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/2/console

Harbormaster failed remote builds in B27472: Diff 26592!Mar 16 2022, 1:32 PM
vlorentz updated this revision to Diff 26593.Mar 16 2022, 1:37 PM

replace unnecessary test data

swh-public-ci added a comment.Mar 16 2022, 1:37 PM

Build has FAILED

Patch application report for D7360 (id=26593)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 5e1dce96bf29bb372f3bb677fbc77c99bb88d816
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/3/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/3/console

Harbormaster failed remote builds in B27473: Diff 26593!Mar 16 2022, 1:37 PM
vlorentz added a reviewer: Reviewers.Mar 16 2022, 2:28 PM
vlorentz updated this revision to Diff 26605.Mar 16 2022, 2:29 PM

add CLI test

vlorentz updated this revision to Diff 26606.Mar 16 2022, 2:29 PM

add missing file

swh-public-ci added a comment.Mar 16 2022, 2:29 PM

Build has FAILED

Patch application report for D7360 (id=26605)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit f982347f75d0d96bafc6c92f4dea6ca6a4dec737
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/4/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/4/console

Harbormaster failed remote builds in B27486: Diff 26605!Mar 16 2022, 2:29 PM
swh-public-ci added a comment.Mar 16 2022, 2:29 PM

Build has FAILED

Patch application report for D7360 (id=26606)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit ccd098c4436b7b09ccde5bd26363d5ff66344565
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/5/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/5/console

Harbormaster failed remote builds in B27487: Diff 26606!Mar 16 2022, 2:30 PM
vlorentz updated this revision to Diff 26608.Mar 16 2022, 2:55 PM

add test with more than 1 corrupt object

swh-public-ci added a comment.Mar 16 2022, 2:56 PM

Build has FAILED

Patch application report for D7360 (id=26608)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 785ee4f78ad45b87da0ace58b487a9046a990ed8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/6/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/6/console

Harbormaster failed remote builds in B27489: Diff 26608!Mar 16 2022, 2:56 PM
ardumont added a subscriber: ardumont.EditedMar 16 2022, 4:07 PM

Ok, i started reviewing this, it's long.
It sounds ok from afar, i stopped at the test_cli so far.

Can you please:

  • land the dependent this diff depends upon (afaict they are accepted now so after you can trigger again the build here).
  • link this diff to the related task

TIA

vlorentz added a task: T3841: regularly scrub all the data stores of swh.Mar 16 2022, 4:08 PM
vlorentz added a comment.EditedMar 16 2022, 4:11 PM

waiting for jenkins to release swh-storage

swh-public-ci added a comment.Mar 16 2022, 6:59 PM

Build has FAILED

Patch application report for D7360 (id=26608)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 785ee4f78ad45b87da0ace58b487a9046a990ed8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/7/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/7/console

swh-public-ci added a comment.Mar 17 2022, 1:27 PM

Build has FAILED

Patch application report for D7360 (id=26608)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 785ee4f78ad45b87da0ace58b487a9046a990ed8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/8/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/8/console

swh-public-ci added a comment.Mar 17 2022, 1:40 PM

Build has FAILED

Patch application report for D7360 (id=26608)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 785ee4f78ad45b87da0ace58b487a9046a990ed8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/9/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/9/console

vlorentz updated this revision to Diff 26636.Mar 17 2022, 1:45 PM

wat

swh-public-ci added a comment.Mar 17 2022, 1:46 PM

Build has FAILED

Patch application report for D7360 (id=26636)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit a8d2533f341b08ed5cbaf93fe1243e4d52e282c3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/10/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/10/console

Harbormaster failed remote builds in B27519: Diff 26636!Mar 17 2022, 1:46 PM
vlorentz updated this revision to Diff 26637.Mar 17 2022, 1:57 PM

fix yaml req

swh-public-ci added a comment.Mar 17 2022, 1:58 PM

Build has FAILED

Patch application report for D7360 (id=26637)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit 72bcf77907d3c481e38a228b8997ef8a000db076
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

Link to build: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/11/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/11/console

Harbormaster failed remote builds in B27520: Diff 26637!Mar 17 2022, 1:58 PM
vlorentz updated this revision to Diff 26638.Mar 17 2022, 2:03 PM

add types-pyyaml

swh-public-ci added a comment.Mar 17 2022, 2:05 PM

Build is green

Patch application report for D7360 (id=26638)

Rebasing onto 5e030a65a9...

Current branch diff-target is up to date.
Changes applied before test
commit be9a35c0c3977cb2b2f4ad2e9378d9cb394611f5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 16 12:33:37 2022 +0100

    Initialize DB schema and postgresql storage checker

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/12/ for more details.

Harbormaster completed remote builds in B27521: Diff 26638.Mar 17 2022, 2:05 PM
vlorentz added a child revision: D7368: Add a journal checker.Mar 17 2022, 3:14 PM
ardumont accepted this revision.Mar 17 2022, 3:34 PM

lgtm

couple of questions.

corrupted feels more correct than 'corrupt' (even though a quick survey and it looks to be ok in english too...)

swh/scrubber/db.py
25

?

swh/scrubber/tests/test_storage_postgresql.py
25

neat, i did not realize we could do that ;)

78

not a big fan of the object_ name...

105

That's actually the same test one on a specific snapshot, another on multiple snapshots.
Is it necessary to keep the first one?

This revision is now accepted and ready to land.Mar 17 2022, 3:34 PM
vlorentz added inline comments.Mar 17 2022, 3:39 PM
swh/scrubber/tests/test_storage_postgresql.py
78

I couldn't think of a better name. obj could work too, but it is discouraged by PEP 8:

If your public attribute name collides with a reserved keyword, append a single trailing underscore to your attribute name. This is preferable to an abbreviation or corrupted spelling.

https://peps.python.org/pep-0008/#designing-for-inheritance

105

the first one checks the checker does not report all objects after the first corrupt one, for example.

Closed by commit rDSCRUBbe9a35c0c397: Initialize DB schema and postgresql storage checker (authored by vlorentz). · Explain WhyMar 22 2022, 11:20 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSCRUBbe9a35c0c397: Initialize DB schema and postgresql storage checker.

Revision Contents
Changeset List

PathSize
1 line
1 line
docs/
34 lines
3 lines
2 lines
swh/
scrubber/
23 lines
91 lines
86 lines
89 lines
sql/
1 line
28 lines
13 lines
tests/
28 lines
64 lines
105 lines

Diff 26608

View Options

README.rst

Loading...
View Options

conftest.py

Loading...
View Options

docs/README.rst

Loading...
View Options

requirements-swh.txt

Loading...
View Options

requirements-test.txt

Loading...
View Options

swh/scrubber/__init__.py

Loading...
View Options

swh/scrubber/bar.py

Loading...
View Options

swh/scrubber/check_storage.py

Loading...
View Options

swh/scrubber/cli.py

Loading...
View Options

swh/scrubber/db.py

Loading...
View Options

swh/scrubber/sql/20-enums.sql

Loading...
View Options

swh/scrubber/sql/30-schema.sql

Loading...
View Options

swh/scrubber/sql/60-indexes.sql

Loading...
View Options

swh/scrubber/tests/conftest.py

Loading...
View Options

swh/scrubber/tests/test_cli.py

Loading...
View Options

swh/scrubber/tests/test_nothing.py

Loading...
View Options

swh/scrubber/tests/test_storage_postgresql.py

Loading...
About · Developer information · Legal