Add checking and reporting of references to missing objects
ClosedPublic
Actions

Authored by vlorentz on Aug 17 2022, 3:39 PM.

Details

Reviewers

anlambert

Group Reviewers

Reviewers

Maniphest Tasks

T4136: Add an "history completeness check"

Commits

rDSCRUB708b2b2ca738: Add checking and reporting of references to missing objects

Summary

Closes T4136.

Diff Detail

Repository

rDSCRUB Datastore Scrubber

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Aug 17 2022, 3:39 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptAug 17 2022, 3:39 PM

Build is green

Patch application report for D8248 (id=29786)

Rebasing onto 3d999c5129...

Current branch diff-target is up to date.

Changes applied before test

commit 08f92de0fc261bf78980c974368963e4d314dfde
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 17 15:38:16 2022 +0200

    Add checking and reporting of references to missing objects

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/50/ for more details.

Harbormaster completed remote builds in B30838: Diff 29786.Aug 17 2022, 3:42 PM

vlorentz requested review of this revision.Aug 17 2022, 3:42 PM

vlorentz edited the summary of this revision. (Show Details)Aug 17 2022, 3:44 PM

vlorentz added a task: T4136: Add an "history completeness check".

Looks good to me.

swh/scrubber/db.py
328	"Yields all objects missing from datastores." seems a better doctring here.
353	"Yields all referenced objects missing from datastores." seems a better doctring here.
swh/scrubber/storage_checker.py
109	missing dot at end of sentence.
186–190	Should this queries be executed in batch in case the number of objects to check is really large ? We do not have any timeouts on those though.

This revision is now accepted and ready to land.Aug 17 2022, 4:31 PM

vlorentz marked 2 inline comments as done.Aug 17 2022, 4:34 PM

vlorentz added inline comments.

swh/scrubber/db.py
328	copy-pasted too fast ;)
swh/scrubber/storage_checker.py
186–190	It's indirectly batched by the length of `objects`, so I'll keep it this way until it becomes an issue

fix docstring typos

Build is green

Patch application report for D8248 (id=29789)

Rebasing onto 3d999c5129...

Current branch diff-target is up to date.

Changes applied before test

commit 708b2b2ca73897c725cb257fdbfc6c430979aa4c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 17 15:38:16 2022 +0200

    Add checking and reporting of references to missing objects

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/51/ for more details.

Harbormaster completed remote builds in B30841: Diff 29789.Aug 17 2022, 4:37 PM

Closed by commit rDSCRUB708b2b2ca738: Add checking and reporting of references to missing objects (authored by vlorentz). · Explain WhyAug 17 2022, 4:37 PM

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDSCRUB708b2b2ca738: Add checking and reporting of references to missing objects.