Page MenuHomeSoftware Heritage

package.debian: Re-implement debian loader within the package loader mechanism
ClosedPublic

Authored by ardumont on Oct 12 2019, 7:58 PM.

Details

Summary

This introduces the debian package loader.

Contrary to other package loaders, a package can have multiple artifacts (per package version).

In those terms, we had to slightly adapt the swh.loader.package.loader.PackageLoader to demonstrate this.
Thus the impacts on pypi, npm, deposit modules as well.

Related T2025
Related D2133
Related D2134

Test Plan

tox

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
package-loader
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 8383
Build 12140: tox-on-jenkinsJenkins
Build 12139: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
  • test_debian: Add test around get_package_metadata function

Diff against package-loader (not master :/)

ardumont edited the summary of this revision. (Show Details)
  • debian.loader: Update typing annotations
  • debian: Explicit we visit twice the same artifacts across 2 visits
  • package.debian: Check for known revisions using snapshot if any

Add test to demonstrate we visit twice the artifacts for 2 visits (with no
change in between)

ardumont retitled this revision from debian: Implement debian loader with package loader mechanism to package.debian: Re-implement debian loader within the package loader mechanism.Oct 15 2019, 2:41 PM
ardumont edited the summary of this revision. (Show Details)
  • debian: Fix edge case about date
  • package.debian: Check for known revisions using snapshot if any
  • package: Manipulate package information instead of artifacts

Build against origin/package-loader!

Build has FAILED

Because when i updated the diff against origin/master, it resetted the build's setup to build against origin/package-loader.
I triggered back the build against the right remote branch...

expected_snapshot = {'branches': {'HEAD': {'target': 'releases/0.0.4', 'target_type': 'alias'}, 'releases/0.0.2': {'target': 'd8a1c7474d29...5acc2239a', 'target_type': 'revision'}}, 'id': b"\xd0X~\x11\x95\xae\xd5\xa8\x80\x04\x11\xa0\x08\xf2\xf2\xd6'\xf1\x8e-"}

something is off in the npm test...
there should not be mixed hex id and bytes id... checking...
somehow that works in current box!

  • fix and add debug statement a bit

Build has FAILED

Ok, the head target for npm is 0.0.2 instead of 0.0.4 in the ci job (it's consistently ok locally :/)
So there is something amiss somewhere.
will check tomorrow.

  • fix and add debug statement in case the snapshot is not found

something is off in the npm test...

and pypi

there should not be mixed hex id and bytes id... checking...

ok that was one pb

somehow that works in current box!

wrongly

we are off in our snapshot branch computations
we no longer use the default_release, fortunately the ci caught it... (1 npm and 1 pypi tests failing for the same reason).

Here is the snapshot for the ci:

{'branches': {b'HEAD': {'target': b'releases/0.0.2', 'target_type': 'alias'},
              b'releases/0.0.2': {'target': b'\xd8\xa1\xc7GM)V\xacY\x8a\x19\xf0'
                                            b'\xf2}R\xf7\x01_\x11~',
                                  'target_type': 'revision'},
              b'releases/0.0.3': {'target': b'_\x9e\xb7\x8a\xf3\x7f\xfd\x12'
                                            b'\x94\x9f#^\x86\xfa\xc0H'
                                            b'\x98\xf9\xf7*',
                                  'target_type': 'revision'},
              b'releases/0.0.4': {'target': b'\xba\x01\x9b\x19+\xdb\x94\xbd'
                                            b'\x0b[\xd6\x8b:_\x92\xb5'
                                            b'\xac\xc2#\x9a',
                                  'target_type': 'revision'}},
 'id': b'b\xaf2\x0e\x106\xbe\xf9\x90FB\x08c\xc1ZD\x87\x0f\xb4%'}

the alias branch targets 0.0.2 but we expect it to be 0.0.4.
Like i said, the default release computation is off now, fixing.

  • package.loader: Fix snapshot branch computations
  • package.*: Unify release name computation
  • debian: No head alias for debian implementation
  • loader.package.tests: Make sure we do not request outside services

Plug to package-loader branch

This revision was not accepted when it landed; it landed in state Needs Review.Oct 17 2019, 1:28 PM
This revision was automatically updated to reflect the committed changes.