Page MenuHomeSoftware Heritage
Differential D1424

algos.revisions_walker: Handle missing revisions in the archive
ClosedPublic
Actions

Authored by anlambert on Apr 18 2019, 4:20 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Commits
rDSTOC6510b5ec4d83: algos.revisions_walker: Handle truncated/shallow histories
rDSTO6510b5ec4d83: algos.revisions_walker: Handle truncated/shallow histories
Summary

It exist rare cases when a revsion is referenced but is missing into the archive content
(some deposit origins have that issue for instance).

So ensure the revisions walker implementation to skip the processing of these missing revisions.

Related T1675

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Apr 18 2019, 4:20 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 18 2019, 4:20 PM
swh-public-ci added a comment.Apr 18 2019, 4:26 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/375/ for more details.

Harbormaster completed remote builds in B5501: Diff 4625.Apr 18 2019, 4:26 PM
olasd requested changes to this revision.Apr 18 2019, 4:31 PM
olasd added a subscriber: olasd.

Please clarify the code and the commit message to say that this is handling truncated/shallow histories where a revision's parent is referenced but doesn't exist in the archive.

Could we find a way to return a flag saying "history truncated", "parent revision not found" or something more explicit than just returning "invalid" results silently?

swh/storage/algos/revisions_walker.py
148

maybe add a comment here saying that the history got truncated?

178

Same here ?

This revision now requires changes to proceed.Apr 18 2019, 4:31 PM
olasd added a comment.Apr 18 2019, 4:59 PM

Just to be clear, the "history truncated" flag stuff can happen later, the new behavior after this minimal change is still better than just crashing

anlambert added a comment.Apr 18 2019, 5:09 PM

Could we find a way to return a flag saying "history truncated", "parent revision not found" or something more explicit than just returning "invalid" results silently?

This indeed would be of interest to have to catch possible loading issues. This needs some specification though, let's create a task on the subject.

swh/storage/algos/revisions_walker.py
148

Sure, will improve that

anlambert mentioned this in T1677: Revisions walker: Add notification for truncated/shallow histories.Apr 18 2019, 5:24 PM

T1677 created for implementing missing revision data notification.

anlambert updated this revision to Diff 4628.Apr 18 2019, 5:34 PM

Update: Address comments

olasd accepted this revision.Apr 18 2019, 5:39 PM
This revision is now accepted and ready to land.Apr 18 2019, 5:39 PM
swh-public-ci added a comment.Apr 18 2019, 5:40 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/376/ for more details.

Harbormaster completed remote builds in B5506: Diff 4628.Apr 18 2019, 5:40 PM
Closed by commit rDSTO6510b5ec4d83: algos.revisions_walker: Handle truncated/shallow histories (authored by anlambert). · Explain WhyApr 18 2019, 5:55 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
swh/
storage/
algos/
17 lines

Diff 4629

View Options

swh/storage/algos/revisions_walker.py

Loading...
About · Developer information · Legal