Page MenuHomeSoftware Heritage
Differential D6608

loader: Implement post_load to check for revision divergence
ClosedPublic
Actions

Authored by ardumont on Nov 4 2021, 4:23 PM.

Details

Reviewers
vlorentz
anlambert
Group Reviewers
Reviewers
Maniphest Tasks
T3695: Investigate revision reconstruction discrepancy with subversion export
Commits
rDLDSVN9d93d8d0fa45: tests: Actually check the last visit is indeed a full visit
rDLDSVN64314742b54a: loader: Implement post_load to check for revision divergence
Summary

When all revisions of a subversion repository have been processed and
loaded, check if the reconstructed filesystem for the last revision
is consistent with the one obtained by a svn export operation.

If it is not, an exception will be raised in order to report the
issue and mark the visit as partial.

Previously that kind of issue could only be detected by triggering
a new load of the repository.

Related to T3695

Diff Detail

Repository
rDLDSVN Subversion (SVN) loader
Branch
diff/24004
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 24922
Build 38941: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 38940: arc lint + arc unit

Event Timeline

anlambert created this revision.Nov 4 2021, 4:23 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 4 2021, 4:23 PM
swh-public-ci added a comment.Nov 4 2021, 4:25 PM

Build is green

Patch application report for D6608 (id=24004)

Rebasing onto 830482755f...

Current branch diff-target is up to date.
Changes applied before test
commit cf9d5c6bcab208e3b28c9ecb5033a7defcf89318
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 4 16:17:16 2021 +0100

    loader: Implement post_load to check for revision divergence
    
    When all revisions of a subversion repository have been processed and
    loaded, check if the reconstructed filesystem for the last revision
    is consistent with the one obtained by a svn export operation.
    
    If it is not, an exception will be raised in order to report the
    issue and mark the visit as partial.
    
    Previously that kind of issue could only be detected by triggering
    a new load of the repository.
    
    Related to T3695

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/159/ for more details.

Harbormaster completed remote builds in B24877: Diff 24004.Nov 4 2021, 4:25 PM
anlambert requested review of this revision.Nov 4 2021, 4:25 PM
vlorentz accepted this revision.Nov 8 2021, 10:48 AM
This revision is now accepted and ready to land.Nov 8 2021, 10:48 AM
ardumont accepted this revision.Nov 8 2021, 11:36 AM
ardumont added a subscriber: ardumont.

good idea.

Thanks.

ardumont commandeered this revision.Nov 8 2021, 4:18 PM
ardumont edited reviewers, added: anlambert; removed: ardumont.
ardumont updated this revision to Diff 24047.Nov 8 2021, 4:50 PM

Rebase on latest master

swh-public-ci added a comment.Nov 8 2021, 4:52 PM

Build has FAILED

Patch application report for D6608 (id=24047)

Rebasing onto 58a07c67d5...

Current branch diff-target is up to date.
Changes applied before test
commit fdafcc9878aa2d3531f7de923d217bcb5e4ab23c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 4 16:17:16 2021 +0100

    loader: Implement post_load to check for revision divergence
    
    When all revisions of a subversion repository have been processed and
    loaded, check if the reconstructed filesystem for the last revision
    is consistent with the one obtained by a svn export operation.
    
    If it is not, an exception will be raised in order to report the
    issue and mark the visit as partial.
    
    Previously that kind of issue could only be detected by triggering
    a new load of the repository.
    
    Related to T3695

Link to build: https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/167/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/167/console

Harbormaster failed remote builds in B24921: Diff 24047!Nov 8 2021, 4:52 PM
ardumont updated this revision to Diff 24048.Nov 8 2021, 5:17 PM

Fix test no longer happy after the rebase introducing the post_load implementation

swh-public-ci added a comment.Nov 8 2021, 5:19 PM

Build is green

Patch application report for D6608 (id=24048)

Rebasing onto 58a07c67d5...

Current branch diff-target is up to date.
Changes applied before test
commit 64314742b54a98af068e9e82049049f8a2d53855
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 4 16:17:16 2021 +0100

    loader: Implement post_load to check for revision divergence
    
    When all revisions of a subversion repository have been processed and
    loaded, check if the reconstructed filesystem for the last revision
    is consistent with the one obtained by a svn export operation.
    
    If it is not, an exception will be raised in order to report the
    issue and mark the visit as partial.
    
    Previously that kind of issue could only be detected by triggering
    a new load of the repository.
    
    Related to T3695

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/168/ for more details.

Harbormaster completed remote builds in B24922: Diff 24048.Nov 8 2021, 5:19 PM
ardumont updated this revision to Diff 24049.Nov 8 2021, 5:24 PM
  • tests: Actually check the last visit is indeed a full visit
swh-public-ci added a comment.Nov 8 2021, 5:25 PM

Build is green

Patch application report for D6608 (id=24049)

Rebasing onto 58a07c67d5...

Current branch diff-target is up to date.
Changes applied before test
commit 9d93d8d0fa452854d571e4acf17aecbd4f0a8c65
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Nov 8 17:22:53 2021 +0100

    tests: Actually check the last visit is indeed a full visit
    
    Prior to this commit, the loader.visit_status() call could be insufficient a check.

commit 64314742b54a98af068e9e82049049f8a2d53855
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 4 16:17:16 2021 +0100

    loader: Implement post_load to check for revision divergence
    
    When all revisions of a subversion repository have been processed and
    loaded, check if the reconstructed filesystem for the last revision
    is consistent with the one obtained by a svn export operation.
    
    If it is not, an exception will be raised in order to report the
    issue and mark the visit as partial.
    
    Previously that kind of issue could only be detected by triggering
    a new load of the repository.
    
    Related to T3695

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/169/ for more details.

Harbormaster completed remote builds in B24923: Diff 24049.Nov 8 2021, 5:25 PM
Closed by commit rDLDSVN64314742b54a: loader: Implement post_load to check for revision divergence (authored by anlambert, committed by ardumont). · Explain WhyNov 8 2021, 5:26 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDLDSVN64314742b54a: loader: Implement post_load to check for revision divergence.
ardumont added a commit: rDLDSVN9d93d8d0fa45: tests: Actually check the last visit is indeed a full visit.
anlambert mentioned this in T3695: Investigate revision reconstruction discrepancy with subversion export.Nov 15 2021, 10:55 AM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
loader/
svn/
14 lines92%
Loading...
tests/
33 lines99%
Loading...

Diff 24048

View Options

swh/loader/svn/loader.py

Loading...
View Options

swh/loader/svn/tests/test_loader.py

Loading...
About · Developer information · Legal