Page MenuHomeSoftware Heritage
Differential D6771

maven: Use the instance base_url as metadata authority URL
ClosedPublic
Actions

Authored by vlorentz on Dec 7 2021, 2:01 PM.

Details

Reviewers
borisbaldassari
ardumont
Group Reviewers
Reviewers
Commits
rDLDBASE98506af074d0: maven: Use the instance base_url as metadata authority URL
Summary

instead of just its netloc, as it is possibly to have multiple maven instances
hosted under the same domain but at different paths.

The code is also simpler this way.

@borisbaldassari Sounds good?

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Dec 7 2021, 2:01 PM
swh-public-ci added a comment.Dec 7 2021, 2:03 PM

Build has FAILED

Patch application report for D6771 (id=24566)

Could not rebase; Attempt merge onto 79b1075e1d...

Updating 79b1075..62f66eb
Fast-forward
 requirements.txt                             |   1 +
 swh/loader/package/maven/loader.py           | 135 ++++++++++++---------------
 swh/loader/package/maven/tests/test_maven.py |   8 +-
 3 files changed, 65 insertions(+), 79 deletions(-)
Changes applied before test
commit 62f66eb83fbeb4e6aea2db6fc178dbf616b2f1e3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 14:01:06 2021 +0100

    maven: Use the instance base_url as metadata authority URL
    
    instead of just its netloc, as it is possibly to have multiple maven instances
    hosted under the same domain but at different paths.
    
    The code is also simpler this way.

commit a96389f5b916b307141aaceac6f8a49e43ca389b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:43:08 2021 +0100

    maven: Don't carry deleted versions over to the next snapshot
    
    Snapshots should only record versions that currently exist;
    even if they used to exist in a previous visits.
    
    If readers of the archive want to access deleted versions,
    than can look up older snapshots.

commit e8b6ed5ab223de3839a3c02d771364207d47160a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:37:17 2021 +0100

    maven: Make MavenPackageInfo.from_metadata more concise

commit 5da115b6e5bf48e3829c830c9f164bd07ed14509
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:34:29 2021 +0100

    maven: Simplify definition of the 'version_artifact' dict
    
    We don't need it to be ordered; and '.keys()' is redundant.

commit ccf71383c61d642f717256a5ed55539073fd0477
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:32:26 2021 +0100

    maven: Simplify build_extrinsic_directory_metadata.

commit a76ab28824a2c203b8cc5f9ff70cecf922770662
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 11:54:33 2021 +0100

    maven: Add typing to the artifacts dict

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/660/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/660/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 7 2021, 2:03 PM
Harbormaster failed remote builds in B25438: Diff 24566!
vlorentz updated this revision to Diff 24567.Dec 7 2021, 2:14 PM

fix crash on empty list of artifacts

vlorentz added a parent revision: D6770: maven: Pass the base URL of the Maven instance to the loader.Dec 7 2021, 2:14 PM
vlorentz updated this revision to Diff 24568.Dec 7 2021, 2:15 PM

slightly cleaner code

swh-public-ci added a comment.Dec 7 2021, 2:16 PM

Build has FAILED

Patch application report for D6771 (id=24567)

Could not rebase; Attempt merge onto 79b1075e1d...

Updating 79b1075..f755b26
Fast-forward
 requirements.txt                             |   1 +
 swh/loader/package/maven/loader.py           | 141 ++++++++++++---------------
 swh/loader/package/maven/tests/test_maven.py |   8 +-
 3 files changed, 71 insertions(+), 79 deletions(-)
Changes applied before test
commit f755b2681a6c5b0cd76f8f46c3ca553279e940f3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 14:01:06 2021 +0100

    maven: Use the instance base_url as metadata authority URL
    
    instead of just its netloc, as it is possibly to have multiple maven instances
    hosted under the same domain but at different paths.
    
    The code is also simpler this way.

commit a96389f5b916b307141aaceac6f8a49e43ca389b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:43:08 2021 +0100

    maven: Don't carry deleted versions over to the next snapshot
    
    Snapshots should only record versions that currently exist;
    even if they used to exist in a previous visits.
    
    If readers of the archive want to access deleted versions,
    than can look up older snapshots.

commit e8b6ed5ab223de3839a3c02d771364207d47160a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:37:17 2021 +0100

    maven: Make MavenPackageInfo.from_metadata more concise

commit 5da115b6e5bf48e3829c830c9f164bd07ed14509
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:34:29 2021 +0100

    maven: Simplify definition of the 'version_artifact' dict
    
    We don't need it to be ordered; and '.keys()' is redundant.

commit ccf71383c61d642f717256a5ed55539073fd0477
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:32:26 2021 +0100

    maven: Simplify build_extrinsic_directory_metadata.

commit a76ab28824a2c203b8cc5f9ff70cecf922770662
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 11:54:33 2021 +0100

    maven: Add typing to the artifacts dict

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/661/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/661/console

Harbormaster failed remote builds in B25439: Diff 24567!Dec 7 2021, 2:16 PM
swh-public-ci added a comment.Dec 7 2021, 2:18 PM

Build has FAILED

Patch application report for D6771 (id=24568)

Could not rebase; Attempt merge onto 79b1075e1d...

Updating 79b1075..ed1cf94
Fast-forward
 requirements.txt                             |   1 +
 swh/loader/package/maven/loader.py           | 141 ++++++++++++---------------
 swh/loader/package/maven/tests/test_maven.py |   8 +-
 3 files changed, 71 insertions(+), 79 deletions(-)
Changes applied before test
commit ed1cf9471bf600bfdfcd19192e9b7f93453a1f75
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 14:01:06 2021 +0100

    maven: Use the instance base_url as metadata authority URL
    
    instead of just its netloc, as it is possibly to have multiple maven instances
    hosted under the same domain but at different paths.
    
    The code is also simpler this way.

commit a96389f5b916b307141aaceac6f8a49e43ca389b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:43:08 2021 +0100

    maven: Don't carry deleted versions over to the next snapshot
    
    Snapshots should only record versions that currently exist;
    even if they used to exist in a previous visits.
    
    If readers of the archive want to access deleted versions,
    than can look up older snapshots.

commit e8b6ed5ab223de3839a3c02d771364207d47160a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:37:17 2021 +0100

    maven: Make MavenPackageInfo.from_metadata more concise

commit 5da115b6e5bf48e3829c830c9f164bd07ed14509
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:34:29 2021 +0100

    maven: Simplify definition of the 'version_artifact' dict
    
    We don't need it to be ordered; and '.keys()' is redundant.

commit ccf71383c61d642f717256a5ed55539073fd0477
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:32:26 2021 +0100

    maven: Simplify build_extrinsic_directory_metadata.

commit a76ab28824a2c203b8cc5f9ff70cecf922770662
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 11:54:33 2021 +0100

    maven: Add typing to the artifacts dict

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/662/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/662/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 7 2021, 2:18 PM
Harbormaster failed remote builds in B25440: Diff 24568!
vlorentz updated this revision to Diff 24569.Dec 7 2021, 2:28 PM

fix test + test name

swh-public-ci added a comment.Dec 7 2021, 2:30 PM

Build is green

Patch application report for D6771 (id=24569)

Could not rebase; Attempt merge onto 79b1075e1d...

Updating 79b1075..972cdc6
Fast-forward
 requirements.txt                             |   1 +
 swh/loader/package/maven/loader.py           | 141 ++++++++++++---------------
 swh/loader/package/maven/tests/test_maven.py |   8 +-
 swh/loader/package/maven/tests/test_tasks.py |   5 +-
 4 files changed, 74 insertions(+), 81 deletions(-)
Changes applied before test
commit 972cdc61fcd8127d827b41fb692b05f6788aac00
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 14:01:06 2021 +0100

    maven: Use the instance base_url as metadata authority URL
    
    instead of just its netloc, as it is possibly to have multiple maven instances
    hosted under the same domain but at different paths.
    
    The code is also simpler this way.

commit a96389f5b916b307141aaceac6f8a49e43ca389b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:43:08 2021 +0100

    maven: Don't carry deleted versions over to the next snapshot
    
    Snapshots should only record versions that currently exist;
    even if they used to exist in a previous visits.
    
    If readers of the archive want to access deleted versions,
    than can look up older snapshots.

commit e8b6ed5ab223de3839a3c02d771364207d47160a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:37:17 2021 +0100

    maven: Make MavenPackageInfo.from_metadata more concise

commit 5da115b6e5bf48e3829c830c9f164bd07ed14509
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:34:29 2021 +0100

    maven: Simplify definition of the 'version_artifact' dict
    
    We don't need it to be ordered; and '.keys()' is redundant.

commit ccf71383c61d642f717256a5ed55539073fd0477
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:32:26 2021 +0100

    maven: Simplify build_extrinsic_directory_metadata.

commit a76ab28824a2c203b8cc5f9ff70cecf922770662
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 11:54:33 2021 +0100

    maven: Add typing to the artifacts dict

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/663/ for more details.

Harbormaster completed remote builds in B25441: Diff 24569.Dec 7 2021, 2:30 PM
vlorentz requested review of this revision.Dec 7 2021, 2:30 PM
borisbaldassari added a comment.Dec 7 2021, 4:23 PM

As said in the lister's corresponding diff, makes sense to me. Thanks for the improvement.

swh/loader/package/maven/loader.py
139–140

ValueError would be thrown only in case of an empty list, right? So why the "more than one maven instance"?

swh/loader/package/maven/tests/test_tasks.py
14

Not sure if the base_url should look like:

According to the lister it should be the latter I think..

vlorentz added inline comments.Dec 7 2021, 4:58 PM
swh/loader/package/maven/loader.py
139–140

It is raised if base_urls does not have exactly one item. But here, the code would not run at all if it is empty, because of the if; so we can be sure it is only raised if there is more than one.

swh/loader/package/maven/tests/test_tasks.py
14

oh you're right, it should be https://repo1.maven.org/maven2/

vlorentz updated this revision to Diff 24596.Dec 7 2021, 5:00 PM

fix typo in test data

swh-public-ci added a comment.Dec 7 2021, 5:01 PM

Build has FAILED

Patch application report for D6771 (id=24596)

Could not rebase; Attempt merge onto e8b6ed5ab2...

Updating e8b6ed5..394fd99
Fast-forward
 swh/loader/package/maven/loader.py           | 35 +++++++++++++++++-----------
 swh/loader/package/maven/tests/test_maven.py |  8 +++++--
 swh/loader/package/maven/tests/test_tasks.py |  5 ++--
 3 files changed, 31 insertions(+), 17 deletions(-)
Changes applied before test
commit 394fd998357bb6c1fbc3abb4ab348ebc811149e4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 14:01:06 2021 +0100

    maven: Use the instance base_url as metadata authority URL
    
    instead of just its netloc, as it is possibly to have multiple maven instances
    hosted under the same domain but at different paths.
    
    The code is also simpler this way.

commit a96389f5b916b307141aaceac6f8a49e43ca389b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 13:43:08 2021 +0100

    maven: Don't carry deleted versions over to the next snapshot
    
    Snapshots should only record versions that currently exist;
    even if they used to exist in a previous visits.
    
    If readers of the archive want to access deleted versions,
    than can look up older snapshots.

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/664/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/664/console

Harbormaster failed remote builds in B25457: Diff 24596!Dec 7 2021, 5:01 PM
ardumont accepted this revision.Dec 8 2021, 11:11 AM
ardumont added a subscriber: ardumont.

lgtm

Remains to fix the build ;)

This revision is now accepted and ready to land.Dec 8 2021, 11:11 AM
vlorentz added a comment.Dec 8 2021, 11:21 AM

wtf is that build failure?

swh-public-ci added a comment.Dec 8 2021, 11:24 AM

Build is green

Patch application report for D6771 (id=24596)

Rebasing onto a96389f5b9...

Current branch diff-target is up to date.
Changes applied before test
commit 394fd998357bb6c1fbc3abb4ab348ebc811149e4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 14:01:06 2021 +0100

    maven: Use the instance base_url as metadata authority URL
    
    instead of just its netloc, as it is possibly to have multiple maven instances
    hosted under the same domain but at different paths.
    
    The code is also simpler this way.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/665/ for more details.

Harbormaster completed remote builds in B25457: Diff 24596.Dec 8 2021, 11:24 AM
ardumont added inline comments.Dec 8 2021, 11:29 AM
swh/loader/package/maven/loader.py
139

i'm gonna pull a "val", you are missing a branch coverage here.

vlorentz updated this revision to Diff 24614.Dec 8 2021, 11:48 AM

add test

vlorentz marked 3 inline comments as done.Dec 8 2021, 11:49 AM
swh-public-ci added a comment.Dec 8 2021, 11:51 AM

Build is green

Patch application report for D6771 (id=24614)

Rebasing onto a96389f5b9...

Current branch diff-target is up to date.
Changes applied before test
commit 98506af074d00cf735c9563fbfb0f5b20cef768a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Dec 7 14:01:06 2021 +0100

    maven: Use the instance base_url as metadata authority URL
    
    instead of just its netloc, as it is possibly to have multiple maven instances
    hosted under the same domain but at different paths.
    
    The code is also simpler this way.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/666/ for more details.

Harbormaster completed remote builds in B25473: Diff 24614.Dec 8 2021, 11:51 AM
Closed by commit rDLDBASE98506af074d0: maven: Use the instance base_url as metadata authority URL (authored by vlorentz). · Explain WhyDec 8 2021, 11:54 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDLDBASE98506af074d0: maven: Use the instance base_url as metadata authority URL.

Revision Contents
Changeset List

PathSize
swh/
loader/
package/
maven/
29 lines
tests/
25 lines
5 lines

Diff 24616

View Options

swh/loader/package/maven/loader.py

Loading...
View Options

swh/loader/package/maven/tests/test_maven.py

Loading...
View Options

swh/loader/package/maven/tests/test_tasks.py

Loading...
About · Developer information · Legal