Details

Reviewers

ardumont
olasd

Group Reviewers

Reviewers

Commits

rDLDBASEfbc8ee1478e3: Fix tests when run by gbp on Sid.
rDLDBASE2ab367ba7cbe: Package loader: Uniformize author and message
rDLDBASE021880d03045: Package loaders: Add a newline at the end of the message

Summary

Since we are going to switch to releases, this looks like the right
time to make some changes in the objects written by package loaders
to make them consistent with each other.

Authors: use the empty string '' instead of placeholders
Message: use the same message format (inspired by the Debian loader)
for all loaders, instead of the empty string / the version /
something else; except for PyPI and Deposit (which have a better
format because we have more metadata available).

Additionally, this commit adds test of each release object,
instead of only relying on its hash.

Depends on D6625, D6618, D6627, D6628, D6626

Test Plan

Tests will fail because this depends on diffs in swh-listers, ~~swh-model, and swh-storage~~ (in addition to other diffs in the same repo)

Diff Detail

Repository

rDLDBASE Generic VCS/Package Loader

Branch

releases-T3636

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 25049
Build 39141: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 39140: arc lint + arc unit

Event Timeline

vlorentz created this revision.Nov 9 2021, 4:32 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptNov 9 2021, 4:32 PM

vlorentz published this revision for review.Nov 9 2021, 4:32 PM

vlorentz edited the test plan for this revision. (Show Details)

Build has FAILED

Patch application report for D6629 (id=24084)

Could not rebase; Attempt merge onto 5063082e7d...

Updating 5063082..255a457
Fast-forward
 docs/index.rst                                   |   3 +-
 docs/package-loader-specifications.rst           | 112 +++++++
 docs/package-loader-tutorial.rst                 |  25 ++
 swh/loader/package/archive/loader.py             |  34 +--
 swh/loader/package/archive/tests/test_archive.py |  76 +++--
 swh/loader/package/cran/loader.py                |  32 +-
 swh/loader/package/cran/tests/test_cran.py       |  97 ++++--
 swh/loader/package/debian/loader.py              |  46 ++-
 swh/loader/package/debian/tests/test_debian.py   | 147 +++++++---
 swh/loader/package/deposit/loader.py             |  52 ++--
 swh/loader/package/deposit/tests/test_deposit.py | 134 ++++++---
 swh/loader/package/loader.py                     | 226 +++++++++-----
 swh/loader/package/nixguix/loader.py             |  33 ++-
 swh/loader/package/nixguix/tests/test_nixguix.py |  98 +++++--
 swh/loader/package/npm/loader.py                 |  30 +-
 swh/loader/package/npm/tests/test_npm.py         | 169 ++++++-----
 swh/loader/package/opam/loader.py                |  90 ++++--
 swh/loader/package/opam/tests/test_opam.py       | 209 +++++++++++--
 swh/loader/package/pypi/loader.py                |  48 +--
 swh/loader/package/pypi/tests/test_pypi.py       | 358 +++++++++--------------
 swh/loader/package/tests/test_loader.py          | 268 ++++++++++++-----
 swh/loader/package/tests/test_loader_metadata.py |  38 ++-
 swh/loader/tests/__init__.py                     |  18 +-
 swh/loader/tests/test_init.py                    |   4 +-
 24 files changed, 1525 insertions(+), 822 deletions(-)
 create mode 100644 docs/package-loader-specifications.rst

Changes applied before test

commit 255a457636fa60831f57d2fb5c44a6f55e351a61
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 9 16:28:59 2021 +0100

    Package loader: Uniformize author and message
    
    Authors: use None instead of the empty string '' or a placeholder
    Message: use the same message format (inspired by the Debian loader)
     for all loaders, instead of the empty string / the version /
     something else; except for PyPI and Deposit (which have a better
     format because we have more metadata available).
    
    Additionally, this commit adds test of each release object,
    instead of only relying on its hash.

commit 059c71b673c28f35f2834f03742d92bc8cae731d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 9 14:17:33 2021 +0100

    Refactor package loaders to make the version part of BasePackageInfo
    
    Half the loaders already had a version field in their PackageInfo class;
    and the version name needed to be passed almost everywhere p_info already did.
    
    This removes some duplication and inconsistencies between loaders.

commit 2af673b617c5d993127c37a2831202ff478f2135
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 9 13:37:31 2021 +0100

    Refactor package loaders to remove temporary revision objects

commit 05496787809354552ea9ed0043d687acabf451d0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Nov 8 13:39:17 2021 +0100

    Document how each package loader populates fields.

commit 89417bb072aade6b68402dbc10eaee61d98e9ae4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Nov 5 13:18:01 2021 +0100

    Make package loaders write releases instead of revisions
    
    The artifacts they load match the semantics of a Release, but we used Revisions
    so far because of technical details (we needed the 'metadata' field of Revision
    that Release lacks) that is no longer relevant (thanks to the metadata storage).
    
    Packages that were loaded by previous versions of the package loader (as revs)
    will be converted to releases. In order to avoid fetching them from the origin,
    the loader will look for an existing extid pointing to a revision (like it used
    to), fetch that revision, extract some fields (directory id, author, date, ...)
    and build a new release using this information.
    
    This commit is unfortunately very large because of all changes in tests, mostly
    just new hashes and renaming 'revision' to 'release' (and various abbreviations
    and capitalizations).
    
    The only meaningful changes are in swh/loader/package/tests/test_loader.py and
    swh/loader/package/loader.py.
    
    To keep this commit as short as possible, I did not yet change individual loaders
    to create releases: they still create revisions, but are converted by the base
    loader. The next commit will refactor them to remove this conversion layer.

commit c0a98a5c4cfe4beac5c0b03bf61459d244b6f132
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 17:58:37 2021 +0100

    tests: Remove duplicate checks
    
    All the '*_missing' tests are already done automatically by check_snapshot
    (it recursively checks all objects are present in the storage).

commit 2311ad9b365e23d1a3c532aa005e32db46bc06f3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 17:53:20 2021 +0100

    tests: Hide utilities from stack traces
    
    They clutter the test output because pytest prints the whole code
    of the function raising the assertionerror.
    
    With this magic variable, the error is shown as if it was raised
    directly in the caller's body.

commit 551c55ff04774b2c4e50e54f701c514064607f6a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 15:49:45 2021 +0100

    package loaders: Make test failures more helpful
    
    Some tests did the following:
    
    1. build a snapshot
    2. get the snapshot from the storage
    3. compare it with the expected snapshot
    4. get the origin visit from the storage and check it
    
    If the loader built a wrong snapshot, the test fails at step 2,
    and the only information displayed is that the expected snapshot id
    does not exist, which is very unhelpful.
    
    Instead, I reordered them as: 1, 4, 2, 3. This way, if a wrong
    snapshot is build by the loader, it is detected when comparing
    the visit, and pytest shows the two hashes.
    Then, the test can be modified to use the hash that is actually
    generated to show the actual snapshot.
    
    This is consistent with what was already done in the pypi loader.
    
    Additionally, I made the following changes:
    
    1. always check stats last (because a difference in numbers is
       hardly actionable without testing other objects)
    2. add a few more snapshot id checks in visits
    3. deduplicated a hardcoded snapshot id.

commit 89a0bfee48ca1663faddaa08b8fb3f163d5cfc6f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 21 14:35:47 2021 +0200

    deposit: Remove 'parent' deposit
    
    The parent is computed by the deposit as the revision of the latest deposit
    in the same origin before the current one.
    Therefore, it is redundant, as it can be recomputed from metadata
    + revision date.
    
    This is a preliminary change needed to make package loaders produce
    releases instead of revisions, as releases don't have parent relationships

commit aeffe01a2b3918262e4bd715ea52c7d7da27807c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 11:45:36 2021 +0100

    opam: Write package definitions to the extrinsic metadata storage

commit 18bbbae719fc9d165d5b543e54d449f5befc083a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 11:43:53 2021 +0100

    Add missing documentation for `get_metadata_authority`.

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/609/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/609/console

Harbormaster failed remote builds in B24959: Diff 24084!Nov 9 2021, 4:34 PM

vlorentz edited the summary of this revision. (Show Details)Nov 9 2021, 4:34 PM

vlorentz added a parent revision: D6626: cran: Pass the package name to the loader.

Use empty string as author instead of None, it's actually unclear how to make the model handle None authors with non-None dates

vlorentz edited the test plan for this revision. (Show Details)Nov 9 2021, 5:43 PM

vlorentz edited the summary of this revision. (Show Details)

Update commit msg

vlorentz edited the summary of this revision. (Show Details)Nov 9 2021, 5:44 PM

Build is green

Patch application report for D6629 (id=24092)

Could not rebase; Attempt merge onto 5063082e7d...

Updating 5063082..c6a5213
Fast-forward
 docs/index.rst                                   |   3 +-
 docs/package-loader-specifications.rst           | 112 +++++++
 docs/package-loader-tutorial.rst                 |  25 ++
 swh/loader/package/archive/loader.py             |  36 +--
 swh/loader/package/archive/tests/test_archive.py |  77 +++--
 swh/loader/package/cran/loader.py                |  32 +-
 swh/loader/package/cran/tests/test_cran.py       |  97 ++++--
 swh/loader/package/debian/loader.py              |  46 ++-
 swh/loader/package/debian/tests/test_debian.py   | 147 +++++++---
 swh/loader/package/deposit/loader.py             |  52 ++--
 swh/loader/package/deposit/tests/test_deposit.py | 134 ++++++---
 swh/loader/package/loader.py                     | 226 +++++++++-----
 swh/loader/package/nixguix/loader.py             |  29 +-
 swh/loader/package/nixguix/tests/test_nixguix.py |  99 +++++--
 swh/loader/package/npm/loader.py                 |  30 +-
 swh/loader/package/npm/tests/test_npm.py         | 169 ++++++-----
 swh/loader/package/opam/loader.py                |  90 ++++--
 swh/loader/package/opam/tests/test_opam.py       | 209 +++++++++++--
 swh/loader/package/pypi/loader.py                |  48 +--
 swh/loader/package/pypi/tests/test_pypi.py       | 358 +++++++++--------------
 swh/loader/package/tests/test_loader.py          | 268 ++++++++++++-----
 swh/loader/package/tests/test_loader_metadata.py |  38 ++-
 swh/loader/tests/__init__.py                     |  18 +-
 swh/loader/tests/test_init.py                    |   4 +-
 24 files changed, 1526 insertions(+), 821 deletions(-)
 create mode 100644 docs/package-loader-specifications.rst

Changes applied before test

commit c6a52138a387ab509f826cc57744c59a934d4697
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 9 16:28:59 2021 +0100

    Package loader: Uniformize author and message
    
    Authors: use None instead of the empty string '' or a placeholder
    Message: use the same message format (inspired by the Debian loader)
     for all loaders, instead of the empty string / the version /
     something else; except for PyPI and Deposit (which have a better
     format because we have more metadata available).
    
    Additionally, this commit adds test of each release object,
    instead of only relying on its hash.

commit 059c71b673c28f35f2834f03742d92bc8cae731d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 9 14:17:33 2021 +0100

    Refactor package loaders to make the version part of BasePackageInfo
    
    Half the loaders already had a version field in their PackageInfo class;
    and the version name needed to be passed almost everywhere p_info already did.
    
    This removes some duplication and inconsistencies between loaders.

commit 2af673b617c5d993127c37a2831202ff478f2135
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 9 13:37:31 2021 +0100

    Refactor package loaders to remove temporary revision objects

commit 05496787809354552ea9ed0043d687acabf451d0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Nov 8 13:39:17 2021 +0100

    Document how each package loader populates fields.

commit 89417bb072aade6b68402dbc10eaee61d98e9ae4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Nov 5 13:18:01 2021 +0100

    Make package loaders write releases instead of revisions
    
    The artifacts they load match the semantics of a Release, but we used Revisions
    so far because of technical details (we needed the 'metadata' field of Revision
    that Release lacks) that is no longer relevant (thanks to the metadata storage).
    
    Packages that were loaded by previous versions of the package loader (as revs)
    will be converted to releases. In order to avoid fetching them from the origin,
    the loader will look for an existing extid pointing to a revision (like it used
    to), fetch that revision, extract some fields (directory id, author, date, ...)
    and build a new release using this information.
    
    This commit is unfortunately very large because of all changes in tests, mostly
    just new hashes and renaming 'revision' to 'release' (and various abbreviations
    and capitalizations).
    
    The only meaningful changes are in swh/loader/package/tests/test_loader.py and
    swh/loader/package/loader.py.
    
    To keep this commit as short as possible, I did not yet change individual loaders
    to create releases: they still create revisions, but are converted by the base
    loader. The next commit will refactor them to remove this conversion layer.

commit c0a98a5c4cfe4beac5c0b03bf61459d244b6f132
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 17:58:37 2021 +0100

    tests: Remove duplicate checks
    
    All the '*_missing' tests are already done automatically by check_snapshot
    (it recursively checks all objects are present in the storage).

commit 2311ad9b365e23d1a3c532aa005e32db46bc06f3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 17:53:20 2021 +0100

    tests: Hide utilities from stack traces
    
    They clutter the test output because pytest prints the whole code
    of the function raising the assertionerror.
    
    With this magic variable, the error is shown as if it was raised
    directly in the caller's body.

commit 551c55ff04774b2c4e50e54f701c514064607f6a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 15:49:45 2021 +0100

    package loaders: Make test failures more helpful
    
    Some tests did the following:
    
    1. build a snapshot
    2. get the snapshot from the storage
    3. compare it with the expected snapshot
    4. get the origin visit from the storage and check it
    
    If the loader built a wrong snapshot, the test fails at step 2,
    and the only information displayed is that the expected snapshot id
    does not exist, which is very unhelpful.
    
    Instead, I reordered them as: 1, 4, 2, 3. This way, if a wrong
    snapshot is build by the loader, it is detected when comparing
    the visit, and pytest shows the two hashes.
    Then, the test can be modified to use the hash that is actually
    generated to show the actual snapshot.
    
    This is consistent with what was already done in the pypi loader.
    
    Additionally, I made the following changes:
    
    1. always check stats last (because a difference in numbers is
       hardly actionable without testing other objects)
    2. add a few more snapshot id checks in visits
    3. deduplicated a hardcoded snapshot id.

commit 89a0bfee48ca1663faddaa08b8fb3f163d5cfc6f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 21 14:35:47 2021 +0200

    deposit: Remove 'parent' deposit
    
    The parent is computed by the deposit as the revision of the latest deposit
    in the same origin before the current one.
    Therefore, it is redundant, as it can be recomputed from metadata
    + revision date.
    
    This is a preliminary change needed to make package loaders produce
    releases instead of revisions, as releases don't have parent relationships

commit aeffe01a2b3918262e4bd715ea52c7d7da27807c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 11:45:36 2021 +0100

    opam: Write package definitions to the extrinsic metadata storage

commit 18bbbae719fc9d165d5b543e54d449f5befc083a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 11:43:53 2021 +0100

    Add missing documentation for `get_metadata_authority`.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/610/ for more details.

Harbormaster completed remote builds in B24967: Diff 24092.Nov 9 2021, 5:45 PM

Build is green

Patch application report for D6629 (id=24093)

Could not rebase; Attempt merge onto 5063082e7d...

Updating 5063082..a48c957
Fast-forward
 docs/index.rst                                   |   3 +-
 docs/package-loader-specifications.rst           | 112 +++++++
 docs/package-loader-tutorial.rst                 |  25 ++
 swh/loader/package/archive/loader.py             |  36 +--
 swh/loader/package/archive/tests/test_archive.py |  77 +++--
 swh/loader/package/cran/loader.py                |  32 +-
 swh/loader/package/cran/tests/test_cran.py       |  97 ++++--
 swh/loader/package/debian/loader.py              |  46 ++-
 swh/loader/package/debian/tests/test_debian.py   | 147 +++++++---
 swh/loader/package/deposit/loader.py             |  52 ++--
 swh/loader/package/deposit/tests/test_deposit.py | 134 ++++++---
 swh/loader/package/loader.py                     | 226 +++++++++-----
 swh/loader/package/nixguix/loader.py             |  29 +-
 swh/loader/package/nixguix/tests/test_nixguix.py |  99 +++++--
 swh/loader/package/npm/loader.py                 |  30 +-
 swh/loader/package/npm/tests/test_npm.py         | 169 ++++++-----
 swh/loader/package/opam/loader.py                |  90 ++++--
 swh/loader/package/opam/tests/test_opam.py       | 209 +++++++++++--
 swh/loader/package/pypi/loader.py                |  48 +--
 swh/loader/package/pypi/tests/test_pypi.py       | 358 +++++++++--------------
 swh/loader/package/tests/test_loader.py          | 268 ++++++++++++-----
 swh/loader/package/tests/test_loader_metadata.py |  38 ++-
 swh/loader/tests/__init__.py                     |  18 +-
 swh/loader/tests/test_init.py                    |   4 +-
 24 files changed, 1526 insertions(+), 821 deletions(-)
 create mode 100644 docs/package-loader-specifications.rst

Changes applied before test

commit a48c957cd0b71b134832784c3458e1dffc086680
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 9 16:28:59 2021 +0100

    Package loader: Uniformize author and message
    
    Authors: use the empty string '' instead of placeholders
    Message: use the same message format (inspired by the Debian loader)
     for all loaders, instead of the empty string / the version /
     something else; except for PyPI and Deposit (which have a better
     format because we have more metadata available).
    
    Additionally, this commit adds test of each release object,
    instead of only relying on its hash.

commit 059c71b673c28f35f2834f03742d92bc8cae731d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 9 14:17:33 2021 +0100

    Refactor package loaders to make the version part of BasePackageInfo
    
    Half the loaders already had a version field in their PackageInfo class;
    and the version name needed to be passed almost everywhere p_info already did.
    
    This removes some duplication and inconsistencies between loaders.

commit 2af673b617c5d993127c37a2831202ff478f2135
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 9 13:37:31 2021 +0100

    Refactor package loaders to remove temporary revision objects

commit 05496787809354552ea9ed0043d687acabf451d0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Nov 8 13:39:17 2021 +0100

    Document how each package loader populates fields.

commit 89417bb072aade6b68402dbc10eaee61d98e9ae4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Nov 5 13:18:01 2021 +0100

    Make package loaders write releases instead of revisions
    
    The artifacts they load match the semantics of a Release, but we used Revisions
    so far because of technical details (we needed the 'metadata' field of Revision
    that Release lacks) that is no longer relevant (thanks to the metadata storage).
    
    Packages that were loaded by previous versions of the package loader (as revs)
    will be converted to releases. In order to avoid fetching them from the origin,
    the loader will look for an existing extid pointing to a revision (like it used
    to), fetch that revision, extract some fields (directory id, author, date, ...)
    and build a new release using this information.
    
    This commit is unfortunately very large because of all changes in tests, mostly
    just new hashes and renaming 'revision' to 'release' (and various abbreviations
    and capitalizations).
    
    The only meaningful changes are in swh/loader/package/tests/test_loader.py and
    swh/loader/package/loader.py.
    
    To keep this commit as short as possible, I did not yet change individual loaders
    to create releases: they still create revisions, but are converted by the base
    loader. The next commit will refactor them to remove this conversion layer.

commit c0a98a5c4cfe4beac5c0b03bf61459d244b6f132
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 17:58:37 2021 +0100

    tests: Remove duplicate checks
    
    All the '*_missing' tests are already done automatically by check_snapshot
    (it recursively checks all objects are present in the storage).

commit 2311ad9b365e23d1a3c532aa005e32db46bc06f3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 17:53:20 2021 +0100

    tests: Hide utilities from stack traces
    
    They clutter the test output because pytest prints the whole code
    of the function raising the assertionerror.
    
    With this magic variable, the error is shown as if it was raised
    directly in the caller's body.

commit 551c55ff04774b2c4e50e54f701c514064607f6a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 15:49:45 2021 +0100

    package loaders: Make test failures more helpful
    
    Some tests did the following:
    
    1. build a snapshot
    2. get the snapshot from the storage
    3. compare it with the expected snapshot
    4. get the origin visit from the storage and check it
    
    If the loader built a wrong snapshot, the test fails at step 2,
    and the only information displayed is that the expected snapshot id
    does not exist, which is very unhelpful.
    
    Instead, I reordered them as: 1, 4, 2, 3. This way, if a wrong
    snapshot is build by the loader, it is detected when comparing
    the visit, and pytest shows the two hashes.
    Then, the test can be modified to use the hash that is actually
    generated to show the actual snapshot.
    
    This is consistent with what was already done in the pypi loader.
    
    Additionally, I made the following changes:
    
    1. always check stats last (because a difference in numbers is
       hardly actionable without testing other objects)
    2. add a few more snapshot id checks in visits
    3. deduplicated a hardcoded snapshot id.

commit 89a0bfee48ca1663faddaa08b8fb3f163d5cfc6f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 21 14:35:47 2021 +0200

    deposit: Remove 'parent' deposit
    
    The parent is computed by the deposit as the revision of the latest deposit
    in the same origin before the current one.
    Therefore, it is redundant, as it can be recomputed from metadata
    + revision date.
    
    This is a preliminary change needed to make package loaders produce
    releases instead of revisions, as releases don't have parent relationships

commit aeffe01a2b3918262e4bd715ea52c7d7da27807c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 11:45:36 2021 +0100

    opam: Write package definitions to the extrinsic metadata storage

commit 18bbbae719fc9d165d5b543e54d449f5befc083a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 4 11:43:53 2021 +0100

    Add missing documentation for `get_metadata_authority`.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/611/ for more details.

Harbormaster completed remote builds in B24968: Diff 24093.Nov 9 2021, 5:46 PM

Thanks, this looks good!

While we're changing them, should the messages have a trailing newline? I believe this is what git tag generates, for instance.

Great point! Done.

Build is green

Patch application report for D6629 (id=24167)

Rebasing onto 1f1bdad899...

First, rewinding head to replay your work on top of it...
Applying: Package loader: Uniformize author and message
Applying: Package loaders: Add a newline at the end of the message

Changes applied before test

commit e263cdf0aa5c672ab8895a74f7f4e61007d716c8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Nov 17 13:44:25 2021 +0100

    Package loaders: Add a newline at the end of the message
    
    To be consistent with Git.

commit de3a9b8af6071ce6e03e880155be457f17015913
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 9 16:28:59 2021 +0100

    Package loader: Uniformize author and message
    
    Authors: use the empty string '' instead of placeholders
    Message: use the same message format (inspired by the Debian loader)
     for all loaders, instead of the empty string / the version /
     something else; except for PyPI and Deposit (which have a better
     format because we have more metadata available).
    
    Additionally, this commit adds test of each release object,
    instead of only relying on its hash.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/614/ for more details.

Harbormaster completed remote builds in B25048: Diff 24167.Nov 17 2021, 1:47 PM

fix typo

Build is green

Patch application report for D6629 (id=24168)

Rebasing onto 1f1bdad899...

First, rewinding head to replay your work on top of it...
Applying: Package loader: Uniformize author and message
Applying: Package loaders: Add a newline at the end of the message

Changes applied before test

commit 58ee86721f3837338e9b7a54906c2883ffd9b886
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Nov 17 13:44:25 2021 +0100

    Package loaders: Add a newline at the end of the message
    
    To be consistent with Git.

commit 1685c91ba577f600c2bca4a96abdd735a50bb9dd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 9 16:28:59 2021 +0100

    Package loader: Uniformize author and message
    
    Authors: use the empty string '' instead of placeholders
    Message: use the same message format (inspired by the Debian loader)
     for all loaders, instead of the empty string / the version /
     something else; except for PyPI and Deposit (which have a better
     format because we have more metadata available).
    
    Additionally, this commit adds test of each release object,
    instead of only relying on its hash.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/615/ for more details.

Harbormaster completed remote builds in B25049: Diff 24168.Nov 17 2021, 1:58 PM

ardumont accepted this revision.Nov 18 2021, 11:53 AM

ardumont added a subscriber: ardumont.

ardumont added inline comments.

swh/loader/package/cran/tests/test_cran.py
37	we should even drop that id and let the snapshot object computes it.

This revision is now accepted and ready to land.Nov 18 2021, 11:53 AM

vlorentz mentioned this in D6396: Implement maven jar source files loader.Nov 20 2021, 10:50 AM

Thanks a lot!

rebase

This revision was landed with ongoing or failed builds.Nov 22 2021, 11:40 AM

This revision was landed with ongoing or failed builds.

Closed by commit rDLDBASE2ab367ba7cbe: Package loader: Uniformize author and message (authored by vlorentz). · Explain Why

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDLDBASE2ab367ba7cbe: Package loader: Uniformize author and message.

vlorentz added a commit: rDLDBASE021880d03045: Package loaders: Add a newline at the end of the message.

vlorentz added a commit: rDLDBASEfbc8ee1478e3: Fix tests when run by gbp on Sid..

Build is green

Patch application report for D6629 (id=24216)

Rebasing onto 021880d030...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-619-D6629.

Changes applied before test

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/619/ for more details.

Harbormaster completed remote builds in B25093: Diff 24216.Nov 22 2021, 11:43 AM

vlorentz mentioned this in T3638: Make package loaders create releases objects instead of revisions.Nov 26 2021, 4:02 PM

Package loader: Uniformize author and message
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Patch application report for D6629 (id=24084)

Changes applied before test

Patch application report for D6629 (id=24092)

Changes applied before test

Patch application report for D6629 (id=24093)

Changes applied before test

Patch application report for D6629 (id=24167)

Changes applied before test

Patch application report for D6629 (id=24168)

Changes applied before test

Patch application report for D6629 (id=24216)

Changes applied before test

Revision Contents
Changeset List

Diff 24168

docs/package-loader-specifications.rst

swh/loader/package/archive/loader.py

swh/loader/package/archive/tests/test_archive.py

swh/loader/package/cran/loader.py

swh/loader/package/cran/tests/test_cran.py

swh/loader/package/debian/loader.py

swh/loader/package/debian/tests/test_debian.py

swh/loader/package/deposit/loader.py

swh/loader/package/deposit/tests/test_deposit.py

swh/loader/package/loader.py

swh/loader/package/nixguix/loader.py

swh/loader/package/nixguix/tests/test_nixguix.py

swh/loader/package/npm/loader.py

swh/loader/package/npm/tests/test_npm.py

swh/loader/package/opam/loader.py

swh/loader/package/opam/tests/test_opam.py

swh/loader/package/pypi/loader.py

swh/loader/package/pypi/tests/test_pypi.py

swh/loader/package/tests/test_loader.py

Package loader: Uniformize author and messageClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D6629 (id=24084)

Changes applied before test

Patch application report for D6629 (id=24092)

Changes applied before test

Patch application report for D6629 (id=24093)

Changes applied before test

Patch application report for D6629 (id=24167)

Changes applied before test

Patch application report for D6629 (id=24168)

Changes applied before test

Patch application report for D6629 (id=24216)

Changes applied before test

Revision ContentsChangeset List

Diff 24168

docs/package-loader-specifications.rst

swh/loader/package/archive/loader.py

swh/loader/package/archive/tests/test_archive.py

swh/loader/package/cran/loader.py

swh/loader/package/cran/tests/test_cran.py

swh/loader/package/debian/loader.py

swh/loader/package/debian/tests/test_debian.py

swh/loader/package/deposit/loader.py

swh/loader/package/deposit/tests/test_deposit.py

swh/loader/package/loader.py

swh/loader/package/nixguix/loader.py

swh/loader/package/nixguix/tests/test_nixguix.py

swh/loader/package/npm/loader.py

swh/loader/package/npm/tests/test_npm.py

swh/loader/package/opam/loader.py

swh/loader/package/opam/tests/test_opam.py

swh/loader/package/pypi/loader.py

swh/loader/package/pypi/tests/test_pypi.py

swh/loader/package/tests/test_loader.py

Package loader: Uniformize author and message
ClosedPublic
Actions

Revision Contents
Changeset List