Page MenuHomeSoftware Heritage

Make the content checker easy to extend and add an implementation
ClosedPublic

Authored by qcampos on Aug 4 2016, 4:38 PM.

Details

Summary

The integrity checker now depends on an abstract class that contains the
base code that is necessary to perform the check on a storage.

This abstract class can be inherited by concrete classes that will define
the checker behavior when a corrupted or missing content is find
(Note that for now, the checker runs on a local storage so the list of
content to check is made from walking the file system. So a missing
content shouldn't happens. But another version could get this list from
another source that is not the file system).

Also, add an checker that update archive database : this implementation
set the status of the archival database corresponding to the real state
of the object. This way, next archival scheduling, the storage may be
used as a destination for a file copy.

Depends on D91
Related T304

Diff Detail

Repository
rDOBJS Object storage
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

qcampos retitled this revision from to Make the content checker extandable and add an implementation.
qcampos edited the test plan for this revision. (Show Details)
qcampos updated this object.
qcampos retitled this revision from Make the content checker extandable and add an implementation to Make the content checker easy to extend and add an implementation.Aug 4 2016, 4:40 PM
qcampos edited edge metadata.
qcampos updated this object.

A few stylistic comments but the gist of it looks fine.

swh/objstorage/checker.py
117 ↗(On Diff #310)

s/have been/is/

184 ↗(On Diff #310)

s/it's/its/

227 ↗(On Diff #310)

No entry for the archive notifier type?

229 ↗(On Diff #310)

Those quotes are wrong (types[checker_type])

qcampos marked 4 inline comments as done.
qcampos edited edge metadata.

Correct some mistakes in docstrings

zack added inline comments.
swh/objstorage/checker.py
16–22 ↗(On Diff #311)

From discussions on other archived diffs, I'm assuming @qcampos will want to switch this to use the abc framework for abstract classes/methods.
If so, I'm looking forward to an updated diff that takes that into account.
If not, let me know, as the rest of this diff can go in as is.

zack added a reviewer: zack.
This revision is now accepted and ready to land.Aug 10 2016, 11:06 AM
qcampos edited edge metadata.

Make the BaseContentChecker class an Abtract Base Class

This revision now requires review to proceed.Aug 10 2016, 1:49 PM
olasd added a reviewer: olasd.
This revision is now accepted and ready to land.Aug 11 2016, 9:00 PM
This revision was automatically updated to reflect the committed changes.
qcampos marked an inline comment as done.