Page MenuHomeSoftware Heritage
Differential D5285

debian: Make resolve_revision_from use the sha256 of the .dsc
ClosedPublic
Actions

Authored by vlorentz on Mar 19 2021, 2:21 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T3141: Make loaders pass ExtId objects between functions instead of "known_artifact"
Commits
rDLDBASE268f83e943a4: debian: Make resolve_revision_from use the sha256 of the .dsc
Summary

Instead of the sha256 + name + ... of all the files of a package.

This will be needed to transition to ExtID, as we can't reasonably write
this large set in the ExtID storage; and the sha256 of the .dsc is good
enough, as the .dsc contains hashes and names of other files.

Diff Detail

Unit TestsFailed
View All

TimeTest
398 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.package.deposit.tests.test_deposit::test_deposit_loading_ok_2
swh_storage = <swh.storage.retry.RetryingProxyStorage object at 0x7f2612913940> deposit_client = <swh.loader.package.deposit.loader.ApiClient object at 0x7f260de13f28> requests_mock_datadir = <requests_mock.mocker.Mocker object at 0x7f260de13710>
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.core.tests.test_converters::test_content_for_storage_data
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.core.tests.test_converters::test_content_for_storage_path
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.core.tests.test_converters::test_content_for_storage_too_long
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.core.tests.test_converters::test_prepare_contents
View Full Test Results (1 Failed · 158 Passed)

Event Timeline

vlorentz created this revision.Mar 19 2021, 2:21 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMar 19 2021, 2:21 PM
swh-public-ci added a comment.Mar 19 2021, 2:23 PM

Build is green

Patch application report for D5285 (id=18932)

Could not rebase; Attempt merge onto 132522e42e...

Merge made by the 'recursive' strategy.
 swh/loader/package/debian/loader.py              | 44 ++++++++++---------
 swh/loader/package/debian/tests/test_debian.py   | 55 ++++++++++++++++++++++++
 swh/loader/package/nixguix/tests/test_nixguix.py | 33 ++++++++------
 3 files changed, 100 insertions(+), 32 deletions(-)
Changes applied before test
commit e29c3d05b974020dcfdcec2da04e3b47ba75e7e5
Merge: 132522e 9a4991b
Author: Jenkins user <jenkins@localhost>
Date:   Fri Mar 19 13:21:22 2021 +0000

    Merge branch 'diff-target' into HEAD

commit 9a4991b391e6b825f9dee3008a410650d434ca6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 17 10:02:21 2021 +0100

    debian: Make resolve_revision_from use the sha256 of the .dsc
    
    Instead of the sha256 + name + ... of all the files of a package.
    
    This will be needed to transition to ExtID, as we can't reasonably write
    this large set in the ExtID storage; and the sha256 of the .dsc is good
    enough, as the .dsc contains hashes and names of other files.

commit 455e05837dc2fb59b4adcf01df27c6f7d64781dd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 16 14:44:33 2021 +0100

    test_load_nixguix_one_common_artifact_from_other_loader: Test all log lines, not just the last one
    
    It's stricter and more readable.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/403/ for more details.

Harbormaster completed remote builds in B20012: Diff 18932.Mar 19 2021, 2:23 PM
vlorentz requested review of this revision.Mar 19 2021, 2:23 PM
vlorentz added a task: T3141: Make loaders pass ExtId objects between functions instead of "known_artifact".Mar 19 2021, 2:48 PM
ardumont added a subscriber: ardumont.Mar 20 2021, 10:06 AM
ardumont added inline comments.
swh/loader/package/debian/tests/test_debian.py
553

It's not immediately clear what's the difference between this test and the previous one.
Can you please clarify intents with a docstring for that test?

ardumont added inline comments.Mar 20 2021, 10:10 AM
swh/loader/package/debian/tests/test_debian.py
501

why corrupt by the way?

and that should be corrupted, shouldn't it?

ardumont requested changes to this revision.Mar 20 2021, 10:11 AM

lgtm

required changes for the sake of clarifying intents in test and one more question.

This revision now requires changes to proceed.Mar 20 2021, 10:11 AM
vlorentz added inline comments.Mar 20 2021, 11:40 AM
swh/loader/package/debian/tests/test_debian.py
501

Because there should only be one .dsc per package. And "corrupt" and "corrupted" are synonyms

553

One is for the known artifacts, and one is for the new artifact

ardumont accepted this revision.Mar 22 2021, 9:17 AM
This revision is now accepted and ready to land.Mar 22 2021, 9:17 AM
vlorentz updated this revision to Diff 18996.Mar 23 2021, 10:20 AM

rebase

swh-public-ci added a comment.Mar 23 2021, 10:23 AM

Build has FAILED

Patch application report for D5285 (id=18996)

Could not rebase; Attempt merge onto 132522e42e...

Updating 132522e..4960aa3
Fast-forward
 swh/loader/package/debian/loader.py              | 44 ++++++++++---------
 swh/loader/package/debian/tests/test_debian.py   | 55 ++++++++++++++++++++++++
 swh/loader/package/nixguix/tests/test_nixguix.py | 33 ++++++++------
 3 files changed, 100 insertions(+), 32 deletions(-)
Changes applied before test
commit 4960aa38b10b29d249b28b92e78e0308f482e551
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 17 10:02:21 2021 +0100

    debian: Make resolve_revision_from use the sha256 of the .dsc
    
    Instead of the sha256 + name + ... of all the files of a package.
    
    This will be needed to transition to ExtID, as we can't reasonably write
    this large set in the ExtID storage; and the sha256 of the .dsc is good
    enough, as the .dsc contains hashes and names of other files.

commit cf8fc9e9bd19b4f303b33ddeb73dc285d835bfcc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 16 14:44:33 2021 +0100

    test_load_nixguix_one_common_artifact_from_other_loader: Test all log lines, not just the last one
    
    It's stricter and more readable.

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/411/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/411/console

Harbormaster failed remote builds in B20073: Diff 18996!Mar 23 2021, 10:23 AM
vlorentz updated this revision to Diff 19023.Mar 23 2021, 12:27 PM

rebase

swh-public-ci added a comment.Mar 23 2021, 12:29 PM

Build is green

Patch application report for D5285 (id=19023)

Rebasing onto 9a339e981a...

Current branch diff-target is up to date.
Changes applied before test
commit 268f83e943a439958840b4d8cdcff5c9eb16429e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 17 10:02:21 2021 +0100

    debian: Make resolve_revision_from use the sha256 of the .dsc
    
    Instead of the sha256 + name + ... of all the files of a package.
    
    This will be needed to transition to ExtID, as we can't reasonably write
    this large set in the ExtID storage; and the sha256 of the .dsc is good
    enough, as the .dsc contains hashes and names of other files.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/423/ for more details.

Harbormaster completed remote builds in B20099: Diff 19023.Mar 23 2021, 12:29 PM
Closed by commit rDLDBASE268f83e943a4: debian: Make resolve_revision_from use the sha256 of the .dsc (authored by vlorentz). · Explain WhyMar 23 2021, 4:33 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDLDBASE268f83e943a4: debian: Make resolve_revision_from use the sha256 of the .dsc.

Revision Contents
Changeset List

PathSize
swh/
loader/
package/
debian/
44 lines
tests/
55 lines

Diff 18996

View Options

swh/loader/package/debian/loader.py

Loading...
View Options

swh/loader/package/debian/tests/test_debian.py

Loading...
About · Developer information · Legal