Page MenuHomeSoftware Heritage
Differential D7092

replay: Handle externals targeting the same path
ClosedPublic
Actions

Authored by anlambert on Feb 4 2022, 5:22 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T611: support for external definitions in the svn/subversion loader
Commits
rDLDSVN20c1445fb11e: replay: Handle externals targeting the same path
Summary

It exists cases where externals defined on a directory target the
same path so ensure to support that edge case.

Related to T611

Fixes some events in SWH-LOADER-SVN-53

Depends on D7091

Diff Detail

Repository
rDLDSVN Subversion (SVN) loader
Branch
externals-with-same-target
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 26551
Build 41527: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 41526: arc lint + arc unit

Event Timeline

anlambert created this revision.Feb 4 2022, 5:22 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 4 2022, 5:22 PM
swh-public-ci added a comment.Feb 4 2022, 5:24 PM

Build is green

Patch application report for D7092 (id=25721)

Could not rebase; Attempt merge onto 09e83fd94c...

Updating 09e83fd..13dccc7
Fast-forward
 requirements-test.txt               |   2 +-
 swh/loader/svn/loader.py            |   6 -
 swh/loader/svn/replay.py            | 300 +++++++++++++++++++++---------------
 swh/loader/svn/svn.py               |   9 +-
 swh/loader/svn/tests/test_loader.py | 109 +++++++++++++
 5 files changed, 293 insertions(+), 133 deletions(-)
Changes applied before test
commit 13dccc7ad24076d99ef722894737ecc6234096d2
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 17:18:47 2022 +0100

    replay: Handle externals targeting the same path
    
    It exists cases where externals defined on a directory target the
    same path so ensure to support that edge case.
    
    Related to T611

commit cb1b59ca340132dd265784611ede974ca43adaf9
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 14:53:42 2022 +0100

    replay: Prevent erroneous removal of versioned path
    
    When processing an external, the content of the target directory is
    first removed before copying external dirs and files.
    
    But that target directory might contain versioned paths so we must
    ensure to restore them after removing the directory content.
    
    Related to T611

commit 651bb4e0b5c42ac67a8c29b2da9986261e9ff682
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 14:27:36 2022 +0100

    replay: Improve signature of DirEditor.open_file method
    
    Add path parameter for consistency with other methods.

commit d5bdea55bc527a94d9e735648a2b835ea2510fba
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 14:07:57 2022 +0100

    replay: Move external processing in a dedicated method
    
    That code grew a lot so better putting it in a dedicated method, plus
    it enables to remove one indentation level for better readability.

commit 369fbea00ea54b202a08db8541a7e6db42a0cc95
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 13:48:00 2022 +0100

    svn: Add debug log before executing checkout or export operation

commit 8bde0663cb4a72a80212e7bab21a962ddf510e25
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 13:40:52 2022 +0100

    requirements-test: Pin pytest to < 7.0.0
    
    pytest v7.0.0 removed _pytest.tmpdir.TempdirFactory, which is used
    by some of the pytest plugins we use.

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/282/ for more details.

Harbormaster completed remote builds in B26551: Diff 25721.Feb 4 2022, 5:24 PM
anlambert requested review of this revision.Feb 4 2022, 5:24 PM
vlorentz accepted this revision.Feb 4 2022, 5:38 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/loader/svn/replay.py
362

this is getting unwieldy; could you add a docstring to explain the format of what it contains?

401

could/should this be replaced by a DirState?

This revision is now accepted and ready to land.Feb 4 2022, 5:38 PM
anlambert mentioned this in D7086: replay: Move external processing in a dedicated method.Feb 4 2022, 5:52 PM
anlambert added inline comments.Feb 4 2022, 6:26 PM
swh/loader/svn/replay.py
362

sure

401

We might want to add other fields in DirState later unrelated to externals so I prefer to keep it that way.

anlambert updated this revision to Diff 25723.Feb 4 2022, 6:36 PM

Add docstring for externals member of DirState.

Remove not needed list materialization.

anlambert updated this revision to Diff 25724.Feb 4 2022, 6:37 PM

Fix typo.

swh-public-ci added a comment.Feb 4 2022, 6:38 PM

Build is green

Patch application report for D7092 (id=25723)

Could not rebase; Attempt merge onto 09e83fd94c...

Updating 09e83fd..b7f818e
Fast-forward
 requirements-test.txt               |   2 +-
 swh/loader/svn/loader.py            |   6 -
 swh/loader/svn/replay.py            | 303 +++++++++++++++++++++---------------
 swh/loader/svn/svn.py               |   9 +-
 swh/loader/svn/tests/test_loader.py | 109 +++++++++++++
 5 files changed, 295 insertions(+), 134 deletions(-)
Changes applied before test
commit b7f818e1c5b5503b1e25ba9e17c7bb4413c7e77b
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 17:18:47 2022 +0100

    replay: Handle externals targeting the same path
    
    It exists cases where externals defined on a directory target the
    same path so ensure to support that edge case.
    
    Related to T611

commit cb1b59ca340132dd265784611ede974ca43adaf9
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 14:53:42 2022 +0100

    replay: Prevent erroneous removal of versioned path
    
    When processing an external, the content of the target directory is
    first removed before copying external dirs and files.
    
    But that target directory might contain versioned paths so we must
    ensure to restore them after removing the directory content.
    
    Related to T611

commit 651bb4e0b5c42ac67a8c29b2da9986261e9ff682
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 14:27:36 2022 +0100

    replay: Improve signature of DirEditor.open_file method
    
    Add path parameter for consistency with other methods.

commit d5bdea55bc527a94d9e735648a2b835ea2510fba
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 14:07:57 2022 +0100

    replay: Move external processing in a dedicated method
    
    That code grew a lot so better putting it in a dedicated method, plus
    it enables to remove one indentation level for better readability.

commit 369fbea00ea54b202a08db8541a7e6db42a0cc95
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 13:48:00 2022 +0100

    svn: Add debug log before executing checkout or export operation

commit 8bde0663cb4a72a80212e7bab21a962ddf510e25
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 13:40:52 2022 +0100

    requirements-test: Pin pytest to < 7.0.0
    
    pytest v7.0.0 removed _pytest.tmpdir.TempdirFactory, which is used
    by some of the pytest plugins we use.

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/283/ for more details.

Harbormaster completed remote builds in B26553: Diff 25723.Feb 4 2022, 6:38 PM
swh-public-ci added a comment.Feb 4 2022, 6:39 PM

Build is green

Patch application report for D7092 (id=25724)

Could not rebase; Attempt merge onto 09e83fd94c...

Updating 09e83fd..20c1445
Fast-forward
 requirements-test.txt               |   2 +-
 swh/loader/svn/loader.py            |   6 -
 swh/loader/svn/replay.py            | 303 +++++++++++++++++++++---------------
 swh/loader/svn/svn.py               |   9 +-
 swh/loader/svn/tests/test_loader.py | 109 +++++++++++++
 5 files changed, 295 insertions(+), 134 deletions(-)
Changes applied before test
commit 20c1445fb11e24df657b17746b63b4d0d1f5fab2
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 17:18:47 2022 +0100

    replay: Handle externals targeting the same path
    
    It exists cases where externals defined on a directory target the
    same path so ensure to support that edge case.
    
    Related to T611

commit cb1b59ca340132dd265784611ede974ca43adaf9
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 14:53:42 2022 +0100

    replay: Prevent erroneous removal of versioned path
    
    When processing an external, the content of the target directory is
    first removed before copying external dirs and files.
    
    But that target directory might contain versioned paths so we must
    ensure to restore them after removing the directory content.
    
    Related to T611

commit 651bb4e0b5c42ac67a8c29b2da9986261e9ff682
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 14:27:36 2022 +0100

    replay: Improve signature of DirEditor.open_file method
    
    Add path parameter for consistency with other methods.

commit d5bdea55bc527a94d9e735648a2b835ea2510fba
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 14:07:57 2022 +0100

    replay: Move external processing in a dedicated method
    
    That code grew a lot so better putting it in a dedicated method, plus
    it enables to remove one indentation level for better readability.

commit 369fbea00ea54b202a08db8541a7e6db42a0cc95
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 13:48:00 2022 +0100

    svn: Add debug log before executing checkout or export operation

commit 8bde0663cb4a72a80212e7bab21a962ddf510e25
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Feb 4 13:40:52 2022 +0100

    requirements-test: Pin pytest to < 7.0.0
    
    pytest v7.0.0 removed _pytest.tmpdir.TempdirFactory, which is used
    by some of the pytest plugins we use.

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/284/ for more details.

Harbormaster completed remote builds in B26554: Diff 25724.Feb 4 2022, 6:39 PM
Closed by commit rDLDSVN20c1445fb11e: replay: Handle externals targeting the same path (authored by anlambert). · Explain WhyFeb 4 2022, 6:41 PM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDLDSVN20c1445fb11e: replay: Handle externals targeting the same path.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
loader/
svn/
43 lines92%
Loading...
tests/
56 lines99%
Loading...

Diff 25721

View Options

swh/loader/svn/replay.py

Loading...
View Options

swh/loader/svn/tests/test_loader.py

Loading...
About · Developer information · Legal