Page MenuHomeSoftware Heritage

loader: Implement post_load to check for revision divergence
ClosedPublic

Authored by ardumont on Nov 4 2021, 4:23 PM.

Details

Summary

When all revisions of a subversion repository have been processed and
loaded, check if the reconstructed filesystem for the last revision
is consistent with the one obtained by a svn export operation.

If it is not, an exception will be raised in order to report the
issue and mark the visit as partial.

Previously that kind of issue could only be detected by triggering
a new load of the repository.

Related to T3695

Diff Detail

Repository
rDLDSVN Subversion (SVN) loader
Branch
diff/24004
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 24923
Build 38943: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 38942: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D6608 (id=24004)

Rebasing onto 830482755f...

Current branch diff-target is up to date.
Changes applied before test
commit cf9d5c6bcab208e3b28c9ecb5033a7defcf89318
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 4 16:17:16 2021 +0100

    loader: Implement post_load to check for revision divergence
    
    When all revisions of a subversion repository have been processed and
    loaded, check if the reconstructed filesystem for the last revision
    is consistent with the one obtained by a svn export operation.
    
    If it is not, an exception will be raised in order to report the
    issue and mark the visit as partial.
    
    Previously that kind of issue could only be detected by triggering
    a new load of the repository.
    
    Related to T3695

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/159/ for more details.

This revision is now accepted and ready to land.Nov 8 2021, 10:48 AM
ardumont added a subscriber: ardumont.

good idea.

Thanks.

ardumont edited reviewers, added: anlambert; removed: ardumont.

Build has FAILED

Patch application report for D6608 (id=24047)

Rebasing onto 58a07c67d5...

Current branch diff-target is up to date.
Changes applied before test
commit fdafcc9878aa2d3531f7de923d217bcb5e4ab23c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 4 16:17:16 2021 +0100

    loader: Implement post_load to check for revision divergence
    
    When all revisions of a subversion repository have been processed and
    loaded, check if the reconstructed filesystem for the last revision
    is consistent with the one obtained by a svn export operation.
    
    If it is not, an exception will be raised in order to report the
    issue and mark the visit as partial.
    
    Previously that kind of issue could only be detected by triggering
    a new load of the repository.
    
    Related to T3695

Link to build: https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/167/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/167/console

Fix test no longer happy after the rebase introducing the post_load implementation

Build is green

Patch application report for D6608 (id=24048)

Rebasing onto 58a07c67d5...

Current branch diff-target is up to date.
Changes applied before test
commit 64314742b54a98af068e9e82049049f8a2d53855
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 4 16:17:16 2021 +0100

    loader: Implement post_load to check for revision divergence
    
    When all revisions of a subversion repository have been processed and
    loaded, check if the reconstructed filesystem for the last revision
    is consistent with the one obtained by a svn export operation.
    
    If it is not, an exception will be raised in order to report the
    issue and mark the visit as partial.
    
    Previously that kind of issue could only be detected by triggering
    a new load of the repository.
    
    Related to T3695

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/168/ for more details.

  • tests: Actually check the last visit is indeed a full visit

Build is green

Patch application report for D6608 (id=24049)

Rebasing onto 58a07c67d5...

Current branch diff-target is up to date.
Changes applied before test
commit 9d93d8d0fa452854d571e4acf17aecbd4f0a8c65
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Nov 8 17:22:53 2021 +0100

    tests: Actually check the last visit is indeed a full visit
    
    Prior to this commit, the loader.visit_status() call could be insufficient a check.

commit 64314742b54a98af068e9e82049049f8a2d53855
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 4 16:17:16 2021 +0100

    loader: Implement post_load to check for revision divergence
    
    When all revisions of a subversion repository have been processed and
    loaded, check if the reconstructed filesystem for the last revision
    is consistent with the one obtained by a svn export operation.
    
    If it is not, an exception will be raised in order to report the
    issue and mark the visit as partial.
    
    Previously that kind of issue could only be detected by triggering
    a new load of the repository.
    
    Related to T3695

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/169/ for more details.