Page MenuHomeSoftware Heritage
Differential D7099

replay: Use remote repository base URL to export external located in it
ClosedPublic
Actions

Authored by anlambert on Feb 7 2022, 3:31 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T611: support for external definitions in the svn/subversion loader
Commits
rDLDSVN0ea0d1282177: replay: Use remote repository base URL to export external located in it
Summary

Some externals might be located in the same repository we are
currently loading. In that case, replace the external base URL
which corresponds to the origin URL by the remote repository URL.

When we use SvnLoaderFromRemoteDump, that remote repository URL
corresponds to a local repository mounted from a dump file so we
can export the external path in a much faster way without any costly
network requests.

Related to T611

Diff Detail

Repository
rDLDSVN Subversion (SVN) loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Feb 7 2022, 3:31 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 7 2022, 3:31 PM
swh-public-ci added a comment.Feb 7 2022, 3:33 PM

Build is green

Patch application report for D7099 (id=25753)

Rebasing onto 20c1445fb1...

Current branch diff-target is up to date.
Changes applied before test
commit 157b81dc7175fe3b8a34bc004ed2e1904fa7a274
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Feb 7 15:20:17 2022 +0100

    replay: Use remote repository base URL to export external located in it
    
    Some externals might be located in the same repository we are
    currently loading. In that case, replace the external base URL
    which corresponds to the origin URL by the remote repository URL.
    
    When we use SvnLoaderFromRemoteDump, that remote repository URL
    corresponds to a local repository mounted from a dump file so we
    can export the external path in a much faster way without any costly
    network requests.
    
    Related to T611

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/286/ for more details.

Harbormaster completed remote builds in B26582: Diff 25753.Feb 7 2022, 3:33 PM
anlambert requested review of this revision.Feb 7 2022, 3:33 PM
vlorentz added a subscriber: vlorentz.Feb 7 2022, 4:10 PM
vlorentz added inline comments.
swh/loader/svn/replay.py
670

Looks like a false positive if external_url is "https://example.org/foo/" and self.svnrepo.origin_url is "https://example.org/foobar/"

anlambert added inline comments.Feb 7 2022, 4:22 PM
swh/loader/svn/replay.py
670

ah right, I must add the trailing slash when checking prefix in that case, thanks

anlambert updated this revision to Diff 25765.Feb 7 2022, 4:45 PM

Update: Add trailing slash when testing URL prefix.

swh-public-ci added a comment.Feb 7 2022, 4:47 PM

Build is green

Patch application report for D7099 (id=25765)

Rebasing onto 20c1445fb1...

Current branch diff-target is up to date.
Changes applied before test
commit 18b5d31df29bed6c0840328a54a3d06adc62346d
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Feb 7 15:20:17 2022 +0100

    replay: Use remote repository base URL to export external located in it
    
    Some externals might be located in the same repository we are
    currently loading. In that case, replace the external base URL
    which corresponds to the origin URL by the remote repository URL.
    
    When we use SvnLoaderFromRemoteDump, that remote repository URL
    corresponds to a local repository mounted from a dump file so we
    can export the external path in a much faster way without any costly
    network requests.
    
    Related to T611

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/287/ for more details.

Harbormaster completed remote builds in B26604: Diff 25765.Feb 7 2022, 4:47 PM
vlorentz added inline comments.Feb 7 2022, 11:14 PM
swh/loader/svn/replay.py
670

could you add a regression test for this?

anlambert added inline comments.Feb 8 2022, 11:23 AM
swh/loader/svn/replay.py
670

I will try, I should be able to test the exported URL by mocking the svnrepo.client object.

anlambert updated this revision to Diff 25814.Feb 8 2022, 11:26 AM

Update: Rebase and improve debug logging

swh-public-ci added a comment.Feb 8 2022, 11:28 AM

Build is green

Patch application report for D7099 (id=25814)

Rebasing onto d929eebd39...

Current branch diff-target is up to date.
Changes applied before test
commit 1aec51eba758c517f2436ec8c59a3dee37465b10
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Feb 7 15:20:17 2022 +0100

    replay: Use remote repository base URL to export external located in it
    
    Some externals might be located in the same repository we are
    currently loading. In that case, replace the external base URL
    which corresponds to the origin URL by the remote repository URL.
    
    When we use SvnLoaderFromRemoteDump, that remote repository URL
    corresponds to a local repository mounted from a dump file so we
    can export the external path in a much faster way without any costly
    network requests.
    
    Related to T611

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/289/ for more details.

Harbormaster completed remote builds in B26658: Diff 25814.Feb 8 2022, 11:28 AM
anlambert updated this revision to Diff 25825.Feb 8 2022, 1:06 PM

Add non regression test

anlambert added inline comments.Feb 8 2022, 1:08 PM
swh/loader/svn/replay.py
670

I managed to write a test checking export URLs are the expected ones.

swh-public-ci added a comment.Feb 8 2022, 1:08 PM

Build is green

Patch application report for D7099 (id=25825)

Rebasing onto d929eebd39...

Current branch diff-target is up to date.
Changes applied before test
commit 0ea0d1282177a5ca7878ea61c07ffeb0d4676ec1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Feb 7 15:20:17 2022 +0100

    replay: Use remote repository base URL to export external located in it
    
    Some externals might be located in the same repository we are
    currently loading. In that case, replace the external base URL
    which corresponds to the origin URL by the remote repository URL.
    
    When we use SvnLoaderFromRemoteDump, that remote repository URL
    corresponds to a local repository mounted from a dump file so we
    can export the external path in a much faster way without any costly
    network requests.
    
    Related to T611

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/290/ for more details.

Harbormaster completed remote builds in B26669: Diff 25825.Feb 8 2022, 1:08 PM
anlambert added a child revision: D7127: replay: Prevent external path removal when defined on multiple dirs.Feb 8 2022, 6:02 PM
ardumont accepted this revision.Feb 9 2022, 2:03 PM
This revision is now accepted and ready to land.Feb 9 2022, 2:03 PM
Closed by commit rDLDSVN0ea0d1282177: replay: Use remote repository base URL to export external located in it (authored by anlambert). · Explain WhyFeb 9 2022, 3:27 PM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDLDSVN0ea0d1282177: replay: Use remote repository base URL to export external located in it.

Revision Contents
Changeset List

PathSize
swh/
loader/
svn/
24 lines
tests/
78 lines

Diff 25860

View Options

swh/loader/svn/replay.py

Loading...
View Options

swh/loader/svn/tests/test_loader.py

Loading...
About · Developer information · Legal