Page MenuHomeSoftware Heritage

migrate_extrinsic_metadata: guess Debian origins.
ClosedPublic

Authored by vlorentz on Thu, Sep 10, 2:23 PM.

Details

Summary

This works by guessing the package name from the original_artifact data,
then building origins that would match the package name, then filtering
out origins by checking if the revision can be reached from them.

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Thu, Sep 10, 2:23 PM
vlorentz updated this revision to Diff 13761.Thu, Sep 10, 2:24 PM

remove TODO

Build was aborted

Patch application report for D3905 (id=13760)

Rebasing onto d24a1e7732...

Current branch diff-target is up to date.
Changes applied before test
commit 454b690d1f327f27b81bfd99b0b273855cafba29
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/913/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/913/console

Build is green

Patch application report for D3905 (id=13761)

Rebasing onto d24a1e7732...

Current branch diff-target is up to date.
Changes applied before test
commit 711740defe16bfe2d3567ec82473dd8a3cbdfb09
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/914/ for more details.

not reviewed the test yet

swh/storage/migrate_extrinsic_metadata.py
179

is this local variable really necessary?

198

unclear to me why assert this rather than ignore it. Why would you fail on that case?

202

unclear to me if this is not-yet-supported situation or if it's a unexpected one.

212

2 nitpicky points:

  • why not assert on filenames rather than package_names ?
  • I find the package_names[0] easier to read to a better fit at showing the intent of this line of code

but meh

225

why not a list comprehension here?

vlorentz updated this revision to Diff 13830.Fri, Sep 11, 11:40 AM

apply comments

vlorentz updated this revision to Diff 13831.Fri, Sep 11, 11:43 AM

actually commit it

Build is green

Patch application report for D3905 (id=13830)

Rebasing onto d24a1e7732...

Current branch diff-target is up to date.
Changes applied before test
commit 71603aca3f016bbbf100325d153d8bf210f29159
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/919/ for more details.

vlorentz updated this revision to Diff 13833.Fri, Sep 11, 11:45 AM
vlorentz marked 2 inline comments as done.

also remove the restructuring

swh/storage/migrate_extrinsic_metadata.py
198

I prefer to whitelist cases as I see them.

202

unexpected; or I would have added a TODO

212

because there are multiple files (eg. kalgebra_19.12.1-1.dsc, kalgebra_19.12.1.orig.tar.xz, kalgebra_19.12.1-1.debian.tar.xz, and kalgebra_19.12.1.orig.tar.xz.asc), but we expect all of them to have the same name

vlorentz updated this revision to Diff 13834.Fri, Sep 11, 11:50 AM

re-add the destructuring, it's a set so we can't use indexing

Build is green

Patch application report for D3905 (id=13831)

Rebasing onto fd6d72f98e...

First, rewinding head to replay your work on top of it...
Applying: migrate_extrinsic_metadata: guess Debian origins.
Changes applied before test
commit a49d065acc5f2807405bfc00ce73efc1ea024da3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/920/ for more details.

Build has FAILED

Patch application report for D3905 (id=13833)

Rebasing onto fd6d72f98e...

First, rewinding head to replay your work on top of it...
Applying: migrate_extrinsic_metadata: guess Debian origins.
Changes applied before test
commit 3709a910e6a1a723ac59b41745f4b65cdc2713e9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/921/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/921/console

Build was aborted

Patch application report for D3905 (id=13834)

Rebasing onto fd6d72f98e...

First, rewinding head to replay your work on top of it...
Applying: migrate_extrinsic_metadata: guess Debian origins.
Changes applied before test
commit 4c67e41f33777fa097ac0da7f930000d44cca3cc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/922/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/922/console

Build is green

Patch application report for D3905 (id=13834)

Rebasing onto ed55e9c775...

First, rewinding head to replay your work on top of it...
Applying: migrate_extrinsic_metadata: guess Debian origins.
Changes applied before test
commit c881768d5fb1706a551b4a8efa7bfdf253867b9e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/926/ for more details.

Build is green

Patch application report for D3905 (id=13863)

Rebasing onto ed55e9c775...

Current branch diff-target is up to date.
Changes applied before test
commit 265fc387f7b3d5f1a55d136b74fa2ee9b9f11f58
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/927/ for more details.

ardumont accepted this revision.Wed, Sep 16, 4:45 PM
ardumont added a subscriber: ardumont.

lgtm

one joke early on ;)

swh/storage/migrate_extrinsic_metadata.py
175
if status.snapshot is None or status.snapshot in seen_snapshots:
    continue

one could be even be vicious and suggest to put the None snapshot in the seen_snapshots
¯\_(ツ)_/¯ :DDDDDDDD and then

if status.snapshot in seen_snapshots:
    continue
This revision is now accepted and ready to land.Wed, Sep 16, 4:45 PM
This revision was landed with ongoing or failed builds.Wed, Sep 16, 5:02 PM
This revision was automatically updated to reflect the committed changes.

Build is green

Patch application report for D3905 (id=13969)

Rebasing onto 3b781a8a52...

Current branch diff-target is up to date.
Changes applied before test
commit 2ad5600f7a63bed949f5e2057b45949320197619
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 10 14:23:12 2020 +0200

    migrate_extrinsic_metadata: guess Debian origins.
    
    This works by guessing the package name from the original_artifact data,
    then building origins that would match the package name, then filtering
    out origins by checking if the revision can be reached from them.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/939/ for more details.