Page MenuHomeSoftware Heritage

migrate_extrinsic_metadata: guess Debian origins.
ClosedPublic

Authored by vlorentz on Sep 10 2020, 2:23 PM.

Details

Summary

This works by guessing the package name from the original_artifact data,
then building origins that would match the package name, then filtering
out origins by checking if the revision can be reached from them.

Diff Detail

Repository
rDSTO Storage manager
Branch
migrate-debian-origin
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 15058
Build 23207: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 23206: arc lint + arc unit

Unit TestsFailed

TimeTest
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.migrate_extrinsic_metadata.test_debian::test_debian_origins_from_row
def test_debian_origins_from_row(): """Tests debian_origins_from_row on a real example (with some parts omitted, for conciseness)."""
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.migrate_extrinsic_metadata.test_debian::test_debian_origins_from_row__check_revisions
def test_debian_origins_from_row__check_revisions(): """Tests debian_origins_from_row errors when the revision at the head of a branch is a DSC and has no parents
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.migrate_extrinsic_metadata.test_debian::test_debian_origins_from_row__no_result
def test_debian_origins_from_row__no_result(): """Tests debian_origins_from_row when there's no origin, visit, status, snapshot, branch, or matching branch.
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin
View Full Test Results (3 Failed · 748 Passed · 23 Skipped)

Event Timeline

Build was aborted

Patch application report for D3905 (id=13760)

Rebasing onto d24a1e7732...

Current branch diff-target is up to date.
Changes applied before test
commit 454b690d1f327f27b81bfd99b0b273855cafba29
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/913/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/913/console

Build is green

Patch application report for D3905 (id=13761)

Rebasing onto d24a1e7732...

Current branch diff-target is up to date.
Changes applied before test
commit 711740defe16bfe2d3567ec82473dd8a3cbdfb09
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/914/ for more details.

not reviewed the test yet

swh/storage/migrate_extrinsic_metadata.py
178

is this local variable really necessary?

197

unclear to me why assert this rather than ignore it. Why would you fail on that case?

201

unclear to me if this is not-yet-supported situation or if it's a unexpected one.

211

2 nitpicky points:

  • why not assert on filenames rather than package_names ?
  • I find the package_names[0] easier to read to a better fit at showing the intent of this line of code

but meh

224

why not a list comprehension here?

Build is green

Patch application report for D3905 (id=13830)

Rebasing onto d24a1e7732...

Current branch diff-target is up to date.
Changes applied before test
commit 71603aca3f016bbbf100325d153d8bf210f29159
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/919/ for more details.

vlorentz marked 2 inline comments as done.

also remove the restructuring

swh/storage/migrate_extrinsic_metadata.py
197

I prefer to whitelist cases as I see them.

201

unexpected; or I would have added a TODO

211

because there are multiple files (eg. kalgebra_19.12.1-1.dsc, kalgebra_19.12.1.orig.tar.xz, kalgebra_19.12.1-1.debian.tar.xz, and kalgebra_19.12.1.orig.tar.xz.asc), but we expect all of them to have the same name

re-add the destructuring, it's a set so we can't use indexing

Build is green

Patch application report for D3905 (id=13831)

Rebasing onto fd6d72f98e...

First, rewinding head to replay your work on top of it...
Applying: migrate_extrinsic_metadata: guess Debian origins.
Changes applied before test
commit a49d065acc5f2807405bfc00ce73efc1ea024da3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/920/ for more details.

Build has FAILED

Patch application report for D3905 (id=13833)

Rebasing onto fd6d72f98e...

First, rewinding head to replay your work on top of it...
Applying: migrate_extrinsic_metadata: guess Debian origins.
Changes applied before test
commit 3709a910e6a1a723ac59b41745f4b65cdc2713e9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/921/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/921/console

Build was aborted

Patch application report for D3905 (id=13834)

Rebasing onto fd6d72f98e...

First, rewinding head to replay your work on top of it...
Applying: migrate_extrinsic_metadata: guess Debian origins.
Changes applied before test
commit 4c67e41f33777fa097ac0da7f930000d44cca3cc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/922/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/922/console

Build is green

Patch application report for D3905 (id=13834)

Rebasing onto ed55e9c775...

First, rewinding head to replay your work on top of it...
Applying: migrate_extrinsic_metadata: guess Debian origins.
Changes applied before test
commit c881768d5fb1706a551b4a8efa7bfdf253867b9e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/926/ for more details.

Build is green

Patch application report for D3905 (id=13863)

Rebasing onto ed55e9c775...

Current branch diff-target is up to date.
Changes applied before test
commit 265fc387f7b3d5f1a55d136b74fa2ee9b9f11f58
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/927/ for more details.

ardumont added a subscriber: ardumont.

lgtm

one joke early on ;)

swh/storage/migrate_extrinsic_metadata.py
174
if status.snapshot is None or status.snapshot in seen_snapshots:
    continue

one could be even be vicious and suggest to put the None snapshot in the seen_snapshots
¯\_(ツ)_/¯ :DDDDDDDD and then

if status.snapshot in seen_snapshots:
    continue
This revision is now accepted and ready to land.Sep 16 2020, 4:45 PM
This revision was landed with ongoing or failed builds.Sep 16 2020, 5:02 PM
This revision was automatically updated to reflect the committed changes.

Build is green

Patch application report for D3905 (id=13969)

Rebasing onto 3b781a8a52...

Current branch diff-target is up to date.
Changes applied before test
commit 2ad5600f7a63bed949f5e2057b45949320197619
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/939/ for more details.