Page MenuHomeSoftware Heritage

replay: Prevent external path removal when defined on multiple dirs
ClosedPublic

Authored by anlambert on Feb 8 2022, 6:02 PM.

Details

Summary

The same absolute external path can be defined on multiple directories
in the repository.

To prevent removal of an external path while it is still defined in
another directory, we need to track the number of references of the
absolute external path by incrementing / decrementing a counter
when adding / removing an external.

Related to T611

Depends on D7099

Diff Detail

Repository
rDLDSVN Subversion (SVN) loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D7127 (id=25841)

Could not rebase; Attempt merge onto d929eebd39...

Updating d929eeb..c46a3b6
Fast-forward
 swh/loader/svn/replay.py            |  63 +++++++++++----
 swh/loader/svn/tests/test_loader.py | 152 +++++++++++++++++++++++++++++++++++-
 2 files changed, 195 insertions(+), 20 deletions(-)
Changes applied before test
commit c46a3b68579a089703ea324b55370abdca73c007
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 8 15:31:49 2022 +0100

    replay: Prevent external path removal when defined on multiple dirs
    
    The same absolute external path can be defined on multiple directories
    in the repository.
    
    To prevent removal of an external path while it is still defined in
    another directory, we need to track the number of references of the
    absolute external path by incrementing / decrementing a counter
    when adding / removing an external.
    
    Related to T611

commit 0ea0d1282177a5ca7878ea61c07ffeb0d4676ec1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Feb 7 15:20:17 2022 +0100

    replay: Use remote repository base URL to export external located in it
    
    Some externals might be located in the same repository we are
    currently loading. In that case, replace the external base URL
    which corresponds to the origin URL by the remote repository URL.
    
    When we use SvnLoaderFromRemoteDump, that remote repository URL
    corresponds to a local repository mounted from a dump file so we
    can export the external path in a much faster way without any costly
    network requests.
    
    Related to T611

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/291/ for more details.

This revision is now accepted and ready to land.Feb 8 2022, 6:15 PM

Update:

  • ensure to decrement external counter only if the external path exists
  • add commit messages in test

Build is green

Patch application report for D7127 (id=25855)

Could not rebase; Attempt merge onto d929eebd39...

Updating d929eeb..7220edb
Fast-forward
 swh/loader/svn/replay.py            |  67 ++++++++++++----
 swh/loader/svn/tests/test_loader.py | 152 +++++++++++++++++++++++++++++++++++-
 2 files changed, 199 insertions(+), 20 deletions(-)
Changes applied before test
commit 7220edbba0f43ab06dee016aa45fc9d3f0673953
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 8 15:31:49 2022 +0100

    replay: Prevent external path removal when defined on multiple dirs
    
    The same absolute external path can be defined on multiple directories
    in the repository.
    
    To prevent removal of an external path while it is still defined in
    another directory, we need to track the number of references of the
    absolute external path by incrementing / decrementing a counter
    when adding / removing an external.
    
    Related to T611

commit 0ea0d1282177a5ca7878ea61c07ffeb0d4676ec1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Feb 7 15:20:17 2022 +0100

    replay: Use remote repository base URL to export external located in it
    
    Some externals might be located in the same repository we are
    currently loading. In that case, replace the external base URL
    which corresponds to the origin URL by the remote repository URL.
    
    When we use SvnLoaderFromRemoteDump, that remote repository URL
    corresponds to a local repository mounted from a dump file so we
    can export the external path in a much faster way without any costly
    network requests.
    
    Related to T611

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/294/ for more details.