Page MenuHomeSoftware Heritage

utils: Fix hidden path processing in parse_external_definition
ClosedPublic

Authored by anlambert on Feb 9 2022, 12:15 PM.

Details

Summary

str.lstrip takes a list of chars to left strip and do not left strip
a string so if a path starts with a '.' it was erroneously removed.

Related to T611

Depends on D7128

Diff Detail

Repository
rDLDSVN Subversion (SVN) loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D7129 (id=25848)

Could not rebase; Attempt merge onto d929eebd39...

Updating d929eeb..7b6c15c
Fast-forward
 swh/loader/svn/replay.py            |  79 ++++++++++++------
 swh/loader/svn/tests/test_loader.py | 158 +++++++++++++++++++++++++++++++++++-
 swh/loader/svn/tests/test_utils.py  |   6 ++
 swh/loader/svn/utils.py             |   3 +-
 4 files changed, 218 insertions(+), 28 deletions(-)
Changes applied before test
commit 7b6c15c848aaf676a9b4e319ede6c092d4dfca6d
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Feb 9 12:10:10 2022 +0100

    utils: Fix hidden path processing in parse_external_definition
    
    str.lstrip takes a list of chars to left strip and do not left strip
    a string so if a path starts with a '.' it was erroneously removed.
    
    Related to T611

commit 23c92f8e2dfb7c41df9a9187ee12a23438e5b565
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 8 17:31:11 2022 +0100

    replay: Fix external tuple use in process_external method
    
    Some required changes were missed in 20c1445fb11e.
    
    The external tuple created at the beginning of the process_external method
    was missing a third boolean member.
    
    The externals defintions in DirState was also missing that boolean member.
    
    Test checking if an external was already exported was invalid.
    
    Also alias the type of that tuple for better code readability.
    
    Related to T611

commit c46a3b68579a089703ea324b55370abdca73c007
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 8 15:31:49 2022 +0100

    replay: Prevent external path removal when defined on multiple dirs
    
    The same absolute external path can be defined on multiple directories
    in the repository.
    
    To prevent removal of an external path while it is still defined in
    another directory, we need to track the number of references of the
    absolute external path by incrementing / decrementing a counter
    when adding / removing an external.
    
    Related to T611

commit 0ea0d1282177a5ca7878ea61c07ffeb0d4676ec1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Feb 7 15:20:17 2022 +0100

    replay: Use remote repository base URL to export external located in it
    
    Some externals might be located in the same repository we are
    currently loading. In that case, replace the external base URL
    which corresponds to the origin URL by the remote repository URL.
    
    When we use SvnLoaderFromRemoteDump, that remote repository URL
    corresponds to a local repository mounted from a dump file so we
    can export the external path in a much faster way without any costly
    network requests.
    
    Related to T611

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/293/ for more details.

This revision is now accepted and ready to land.Feb 9 2022, 1:00 PM

Build is green

Patch application report for D7129 (id=25857)

Could not rebase; Attempt merge onto d929eebd39...

Updating d929eeb..d7a97a7
Fast-forward
 swh/loader/svn/replay.py            |  83 +++++++++++++------
 swh/loader/svn/tests/test_loader.py | 158 +++++++++++++++++++++++++++++++++++-
 swh/loader/svn/tests/test_utils.py  |   6 ++
 swh/loader/svn/utils.py             |   3 +-
 4 files changed, 222 insertions(+), 28 deletions(-)
Changes applied before test
commit d7a97a75a6575453f1469a13987ece8c685f08d1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Feb 9 12:10:10 2022 +0100

    utils: Fix hidden path processing in parse_external_definition
    
    str.lstrip takes a list of chars to left strip and do not left strip
    a string so if a path starts with a '.' it was erroneously removed.
    
    Related to T611

commit 46f25850ae8e23e004db988abd67ff4841ee0197
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 8 17:31:11 2022 +0100

    replay: Fix external tuple use in process_external method
    
    Some required changes were missed in 20c1445fb11e.
    
    The external tuple created at the beginning of the process_external method
    was missing a third boolean member.
    
    The externals defintions in DirState was also missing that boolean member.
    
    Test checking if an external was already exported was invalid.
    
    Also alias the type of that tuple for better code readability.
    
    Related to T611

commit 7220edbba0f43ab06dee016aa45fc9d3f0673953
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 8 15:31:49 2022 +0100

    replay: Prevent external path removal when defined on multiple dirs
    
    The same absolute external path can be defined on multiple directories
    in the repository.
    
    To prevent removal of an external path while it is still defined in
    another directory, we need to track the number of references of the
    absolute external path by incrementing / decrementing a counter
    when adding / removing an external.
    
    Related to T611

commit 0ea0d1282177a5ca7878ea61c07ffeb0d4676ec1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Feb 7 15:20:17 2022 +0100

    replay: Use remote repository base URL to export external located in it
    
    Some externals might be located in the same repository we are
    currently loading. In that case, replace the external base URL
    which corresponds to the origin URL by the remote repository URL.
    
    When we use SvnLoaderFromRemoteDump, that remote repository URL
    corresponds to a local repository mounted from a dump file so we
    can export the external path in a much faster way without any costly
    network requests.
    
    Related to T611

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/296/ for more details.