Page MenuHomeSoftware Heritage

replay: Fix external tuple use in process_external method
ClosedPublic

Authored by anlambert on Feb 8 2022, 6:03 PM.

Details

Summary

Some required changes were missed in 20c1445fb11e, fortunately that commit
is not yet tagged neither deployed.

The external tuple created at the beginning of the process_external method
was missing a third boolean member.

The externals defintions in DirState was also missing that boolean member.

Test checking if an external was already exported was invalid.

Also alias the type of that tuple for better code readability.

Related to T611

Depends on D7127

Diff Detail

Repository
rDLDSVN Subversion (SVN) loader
Branch
external-tuple-fix
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 26685
Build 41729: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 41728: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D7128 (id=25842)

Could not rebase; Attempt merge onto d929eebd39...

Updating d929eeb..23c92f8
Fast-forward
 swh/loader/svn/replay.py            |  79 ++++++++++++------
 swh/loader/svn/tests/test_loader.py | 158 +++++++++++++++++++++++++++++++++++-
 2 files changed, 210 insertions(+), 27 deletions(-)
Changes applied before test
commit 23c92f8e2dfb7c41df9a9187ee12a23438e5b565
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 8 17:31:11 2022 +0100

    replay: Fix external tuple use in process_external method
    
    Some required changes were missed in 20c1445fb11e.
    
    The external tuple created at the beginning of the process_external method
    was missing a third boolean member.
    
    The externals defintions in DirState was also missing that boolean member.
    
    Test checking if an external was already exported was invalid.
    
    Also alias the type of that tuple for better code readability.
    
    Related to T611

commit c46a3b68579a089703ea324b55370abdca73c007
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 8 15:31:49 2022 +0100

    replay: Prevent external path removal when defined on multiple dirs
    
    The same absolute external path can be defined on multiple directories
    in the repository.
    
    To prevent removal of an external path while it is still defined in
    another directory, we need to track the number of references of the
    absolute external path by incrementing / decrementing a counter
    when adding / removing an external.
    
    Related to T611

commit 0ea0d1282177a5ca7878ea61c07ffeb0d4676ec1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Feb 7 15:20:17 2022 +0100

    replay: Use remote repository base URL to export external located in it
    
    Some externals might be located in the same repository we are
    currently loading. In that case, replace the external base URL
    which corresponds to the origin URL by the remote repository URL.
    
    When we use SvnLoaderFromRemoteDump, that remote repository URL
    corresponds to a local repository mounted from a dump file so we
    can export the external path in a much faster way without any costly
    network requests.
    
    Related to T611

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/292/ for more details.

This revision is now accepted and ready to land.Feb 9 2022, 1:02 PM

Build is green

Patch application report for D7128 (id=25856)

Could not rebase; Attempt merge onto d929eebd39...

Updating d929eeb..46f2585
Fast-forward
 swh/loader/svn/replay.py            |  83 +++++++++++++------
 swh/loader/svn/tests/test_loader.py | 158 +++++++++++++++++++++++++++++++++++-
 2 files changed, 214 insertions(+), 27 deletions(-)
Changes applied before test
commit 46f25850ae8e23e004db988abd67ff4841ee0197
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 8 17:31:11 2022 +0100

    replay: Fix external tuple use in process_external method
    
    Some required changes were missed in 20c1445fb11e.
    
    The external tuple created at the beginning of the process_external method
    was missing a third boolean member.
    
    The externals defintions in DirState was also missing that boolean member.
    
    Test checking if an external was already exported was invalid.
    
    Also alias the type of that tuple for better code readability.
    
    Related to T611

commit 7220edbba0f43ab06dee016aa45fc9d3f0673953
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 8 15:31:49 2022 +0100

    replay: Prevent external path removal when defined on multiple dirs
    
    The same absolute external path can be defined on multiple directories
    in the repository.
    
    To prevent removal of an external path while it is still defined in
    another directory, we need to track the number of references of the
    absolute external path by incrementing / decrementing a counter
    when adding / removing an external.
    
    Related to T611

commit 0ea0d1282177a5ca7878ea61c07ffeb0d4676ec1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Feb 7 15:20:17 2022 +0100

    replay: Use remote repository base URL to export external located in it
    
    Some externals might be located in the same repository we are
    currently loading. In that case, replace the external base URL
    which corresponds to the origin URL by the remote repository URL.
    
    When we use SvnLoaderFromRemoteDump, that remote repository URL
    corresponds to a local repository mounted from a dump file so we
    can export the external path in a much faster way without any costly
    network requests.
    
    Related to T611

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/295/ for more details.