Page MenuHomeSoftware Heritage
Differential D2135

package.debian: Re-implement debian loader within the package loader mechanism
ClosedPublic
Actions

Authored by ardumont on Oct 12 2019, 7:58 PM.

Details

Reviewers
douardda
olasd
Group Reviewers
Reviewers
Commits
rDLDBASE402c3db6bc93: debian: No head alias for debian implementation
rDLDBASE2c3bcb62d8a1: loader.package.tests: Make sure we do not request outside services
rDLDBASE7179007a81bc: package: Manipulate package information instead of artifacts
rDLDBASEcfe6be0793ac: fix and add debug statement in case the snapshot is not found
rDLDBASEa1da93b43a2e: package.loader: Fix snapshot branch computations
rDLDBASEb60abda87dc3: package.*: Unify release name computation
rDLDBASE90d9f8861360: debian: Fix edge case about date
rDLDBASE5389c9c7f2b4: package.debian: Check for known revisions using snapshot if any
rDLDBASE805863b41edd: debian: Explicit we visit twice the same artifacts across 2 visits
rDLDBASEb2db9931144d: debian.loader: Update typing annotations
rDLDBASE65ac1a06b315: test_debian: Add test around get_package_metadata function
rDLDBASE56b2e066cfa6: package.loader: Update comment
rDLDBASEc0e9a8394120: debian.tests: Update pytest_plugin version use
rDLDBASE65b0a7570b61: debian: Expose debian loader implementation first
rDLDBASE3310b012e389: debian: Simplify build revision code
rDLDBASE361e84d127f5: debian: Remove duplicated computations step
rDLDBASE245259adc2e2: package.debian: Compute the dsc information
rDLDBASEc19a63428ba9: package.loader: Clarify download_package contract
rDLDBASE58c532fd62c5: debian: Implement debian loader with package loader mechanism
Summary

This introduces the debian package loader.

Contrary to other package loaders, a package can have multiple artifacts (per package version).

In those terms, we had to slightly adapt the swh.loader.package.loader.PackageLoader to demonstrate this.
Thus the impacts on pypi, npm, deposit modules as well.

Related T2025
Related D2133
Related D2134

Test Plan

tox

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
debian-loader
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 8456
Build 12266: tox-on-jenkinsJenkins
Build 12265: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ardumont updated this revision to Diff 7239.Oct 15 2019, 12:17 PM
  • test_debian: Add test around get_package_metadata function
ardumont updated this revision to Diff 7240.Oct 15 2019, 12:17 PM

Diff against package-loader (not master :/)

swh-public-ci added a comment.Oct 15 2019, 12:18 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/184/ for more details.

Harbormaster completed remote builds in B8456: Diff 7239.Oct 15 2019, 12:18 PM
swh-public-ci added a comment.Oct 15 2019, 12:19 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/185/ for more details.

Harbormaster completed remote builds in B8457: Diff 7240.Oct 15 2019, 12:19 PM
ardumont edited the summary of this revision. (Show Details)Oct 15 2019, 1:10 PM
ardumont edited the summary of this revision. (Show Details)
ardumont updated this revision to Diff 7247.Oct 15 2019, 1:58 PM
  • debian.loader: Update typing annotations
  • debian: Explicit we visit twice the same artifacts across 2 visits
  • package.debian: Check for known revisions using snapshot if any
swh-public-ci added a comment.Oct 15 2019, 1:59 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/186/ for more details.

Harbormaster completed remote builds in B8464: Diff 7247.Oct 15 2019, 1:59 PM
ardumont updated this revision to Diff 7248.Oct 15 2019, 1:59 PM

Add test to demonstrate we visit twice the artifacts for 2 visits (with no
change in between)

swh-public-ci added a comment.Oct 15 2019, 2:00 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/187/ for more details.

Harbormaster completed remote builds in B8465: Diff 7248.Oct 15 2019, 2:00 PM
ardumont added a child revision: D2142: package.debian: Use last snapshot to avoid re-loading a debian package.Oct 15 2019, 2:02 PM
ardumont edited the summary of this revision. (Show Details)Oct 15 2019, 2:08 PM
ardumont retitled this revision from debian: Implement debian loader with package loader mechanism to package.debian: Re-implement debian loader within the package loader mechanism.Oct 15 2019, 2:41 PM
ardumont edited the summary of this revision. (Show Details)
ardumont updated this revision to Diff 7256.Oct 15 2019, 5:06 PM
  • debian: Fix edge case about date
swh-public-ci added a comment.Oct 15 2019, 5:07 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/192/ for more details.

Harbormaster completed remote builds in B8472: Diff 7256.Oct 15 2019, 5:07 PM
ardumont mentioned this in D2142: package.debian: Use last snapshot to avoid re-loading a debian package.Oct 16 2019, 6:37 PM
ardumont updated this revision to Diff 7271.Oct 16 2019, 6:39 PM
  • package.debian: Check for known revisions using snapshot if any
  • package: Manipulate package information instead of artifacts
ardumont updated this revision to Diff 7272.Oct 16 2019, 6:40 PM

Build against origin/package-loader!

swh-public-ci added a comment.Oct 16 2019, 6:40 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/198/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/198/console

Harbormaster failed remote builds in B8487: Diff 7271!Oct 16 2019, 6:40 PM
swh-public-ci added a comment.Oct 16 2019, 6:41 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/199/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/199/console

Harbormaster failed remote builds in B8488: Diff 7272!Oct 16 2019, 6:41 PM
swh-public-ci added a comment.Oct 16 2019, 6:47 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/200/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/200/console

ardumont added a comment.Oct 16 2019, 6:48 PM

Build has FAILED

Because when i updated the diff against origin/master, it resetted the build's setup to build against origin/package-loader.
I triggered back the build against the right remote branch...

swh-public-ci added a comment.Oct 16 2019, 6:51 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/201/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/201/console

ardumont mentioned this in D2145: package.tar: Add a generic archive loader implementation (merge with gnu's).Oct 16 2019, 7:02 PM
ardumont added a child revision: D2145: package.tar: Add a generic archive loader implementation (merge with gnu's).Oct 16 2019, 7:04 PM
ardumont added a comment.Oct 16 2019, 7:12 PM

expected_snapshot = {'branches': {'HEAD': {'target': 'releases/0.0.4', 'target_type': 'alias'}, 'releases/0.0.2': {'target': 'd8a1c7474d29...5acc2239a', 'target_type': 'revision'}}, 'id': b"\xd0X~\x11\x95\xae\xd5\xa8\x80\x04\x11\xa0\x08\xf2\xf2\xd6'\xf1\x8e-"}

something is off in the npm test...
there should not be mixed hex id and bytes id... checking...
somehow that works in current box!

ardumont updated this revision to Diff 7274.Oct 16 2019, 7:14 PM
  • fix (tryout)
swh-public-ci added a comment.Oct 16 2019, 7:15 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/205/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/205/console

Harbormaster failed remote builds in B8490: Diff 7274!Oct 16 2019, 7:15 PM
ardumont updated this revision to Diff 7275.Oct 16 2019, 7:27 PM
  • fix and add debug statement a bit
swh-public-ci added a comment.Oct 16 2019, 7:28 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/206/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/206/console

Harbormaster failed remote builds in B8491: Diff 7275!Oct 16 2019, 7:28 PM
ardumont added a comment.Oct 16 2019, 7:37 PM

Build has FAILED

Ok, the head target for npm is 0.0.2 instead of 0.0.4 in the ci job (it's consistently ok locally :/)
So there is something amiss somewhere.
will check tomorrow.

ardumont updated this revision to Diff 7276.Oct 16 2019, 7:48 PM
  • fix and add debug statement in case the snapshot is not found
swh-public-ci added a comment.Oct 16 2019, 7:49 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/207/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/207/console

Harbormaster failed remote builds in B8492: Diff 7276!Oct 16 2019, 7:49 PM
ardumont edited the summary of this revision. (Show Details)Oct 16 2019, 7:51 PM
ardumont added a comment.Oct 17 2019, 4:29 AM

something is off in the npm test...

and pypi

there should not be mixed hex id and bytes id... checking...

ok that was one pb

somehow that works in current box!

wrongly

we are off in our snapshot branch computations
we no longer use the default_release, fortunately the ci caught it... (1 npm and 1 pypi tests failing for the same reason).

Here is the snapshot for the ci:

{'branches': {b'HEAD': {'target': b'releases/0.0.2', 'target_type': 'alias'},
              b'releases/0.0.2': {'target': b'\xd8\xa1\xc7GM)V\xacY\x8a\x19\xf0'
                                            b'\xf2}R\xf7\x01_\x11~',
                                  'target_type': 'revision'},
              b'releases/0.0.3': {'target': b'_\x9e\xb7\x8a\xf3\x7f\xfd\x12'
                                            b'\x94\x9f#^\x86\xfa\xc0H'
                                            b'\x98\xf9\xf7*',
                                  'target_type': 'revision'},
              b'releases/0.0.4': {'target': b'\xba\x01\x9b\x19+\xdb\x94\xbd'
                                            b'\x0b[\xd6\x8b:_\x92\xb5'
                                            b'\xac\xc2#\x9a',
                                  'target_type': 'revision'}},
 'id': b'b\xaf2\x0e\x106\xbe\xf9\x90FB\x08c\xc1ZD\x87\x0f\xb4%'}

the alias branch targets 0.0.2 but we expect it to be 0.0.4.
Like i said, the default release computation is off now, fixing.

ardumont updated this revision to Diff 7277.Oct 17 2019, 5:03 AM
  • package.loader: Fix snapshot branch computations
  • package.*: Unify release name computation
swh-public-ci added a comment.Oct 17 2019, 5:05 AM

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/208/ for more details.

Harbormaster completed remote builds in B8493: Diff 7277.Oct 17 2019, 5:05 AM
ardumont updated this revision to Diff 7280.Oct 17 2019, 11:39 AM
  • debian: No head alias for debian implementation
  • loader.package.tests: Make sure we do not request outside services
swh-public-ci added a comment.Oct 17 2019, 11:41 AM

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/211/ for more details.

Harbormaster completed remote builds in B8496: Diff 7280.Oct 17 2019, 11:41 AM
ardumont updated this revision to Diff 7283.Oct 17 2019, 1:27 PM

Plug to package-loader branch

swh-public-ci added a comment.Oct 17 2019, 1:28 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/214/ for more details.

Harbormaster completed remote builds in B8499: Diff 7283.Oct 17 2019, 1:28 PM
This revision was not accepted when it landed; it landed in state Needs Review.Oct 17 2019, 1:28 PM
Closed by commit rDLDBASE58c532fd62c5: debian: Implement debian loader with package loader mechanism (authored by ardumont). · Explain Why
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDLDBASE58c532fd62c5: debian: Implement debian loader with package loader mechanism.
ardumont added a commit: rDLDBASEc19a63428ba9: package.loader: Clarify download_package contract.
ardumont added a commit: rDLDBASE245259adc2e2: package.debian: Compute the dsc information.
ardumont added a commit: rDLDBASE361e84d127f5: debian: Remove duplicated computations step.
ardumont added a commit: rDLDBASE3310b012e389: debian: Simplify build revision code.
ardumont added a commit: rDLDBASE65b0a7570b61: debian: Expose debian loader implementation first.
ardumont added a commit: rDLDBASEc0e9a8394120: debian.tests: Update pytest_plugin version use.
ardumont added a commit: rDLDBASE65ac1a06b315: test_debian: Add test around get_package_metadata function.
ardumont added a commit: rDLDBASE56b2e066cfa6: package.loader: Update comment.
ardumont added a commit: rDLDBASEb2db9931144d: debian.loader: Update typing annotations.
ardumont added a commit: rDLDBASE805863b41edd: debian: Explicit we visit twice the same artifacts across 2 visits.
ardumont added a commit: rDLDBASE90d9f8861360: debian: Fix edge case about date.
ardumont added a commit: rDLDBASE5389c9c7f2b4: package.debian: Check for known revisions using snapshot if any.
ardumont added a commit: rDLDBASE7179007a81bc: package: Manipulate package information instead of artifacts.
ardumont added a commit: rDLDBASEcfe6be0793ac: fix and add debug statement in case the snapshot is not found.
ardumont added a commit: rDLDBASEa1da93b43a2e: package.loader: Fix snapshot branch computations.
ardumont added a commit: rDLDBASEb60abda87dc3: package.*: Unify release name computation.
ardumont added a commit: rDLDBASE402c3db6bc93: debian: No head alias for debian implementation.
ardumont added a commit: rDLDBASE2c3bcb62d8a1: loader.package.tests: Make sure we do not request outside services.
ardumont mentioned this in D2155: lister-debian: Install debian model within the lister repository.Oct 17 2019, 6:05 PM
ardumont mentioned this in rDSTO484cebb2c34f: schemata: Send only origin url for scheduling debian loader task.Oct 17 2019, 6:07 PM
ardumont removed a child revision: D2145: package.tar: Add a generic archive loader implementation (merge with gnu's).Oct 17 2019, 6:22 PM
ardumont added a child revision: D2145: package.tar: Add a generic archive loader implementation (merge with gnu's).Oct 18 2019, 7:22 PM
ardumont mentioned this in rDSTOC484cebb2c34f: schemata: Send only origin url for scheduling debian loader task.Oct 30 2019, 5:22 PM
ardumont mentioned this in rDLS56d7cff6e122: debian/model: Install lister model within the lister repository.Nov 4 2019, 10:13 AM
ardumont mentioned this in rDLS7c247c8a4a28: debian/lister: Use url parameter name instead of origin.

Revision Contents
Changeset List

PathSize
1 line
4 lines
1 line
3 lines
swh/
loader/
package/
19 lines
317 lines
153 lines
191 lines
702 lines
298 lines
186 lines
5 lines
tests/
107 lines
14 lines
data/
http_deb.debian.org/
44 lines
1 line
https_deposit.softwareheritage.org/
1 line
1 line
80 lines
https_files.pythonhosted.org/
1 line
1 line
1 line
1 line
1 line
1 line
1 line
https_ftp.gnu.org/
https_pypi.org/
95 lines
114 lines
74 lines
https_registry.npmjs.org/
https_replicate.npmjs.com/
191 lines
347 lines
16 lines
data/
https_ftp.gnu.org/
resources/
tarballs/
resources/
tarballs/
175 lines
246 lines
199 lines
349 lines
526 lines
654 lines
146 lines
105 lines
11 lines

Diff 7239

View Options

MANIFEST.in

Loading...
View Options

requirements-swh.txt

Loading...
View Options

requirements-test.txt

Loading...
View Options

requirements.txt

Loading...
View Options

swh/loader/package/__init__.py

Loading...
View Options

swh/loader/package/debian.py

Loading...
View Options

swh/loader/package/deposit.py

Loading...
View Options

swh/loader/package/gnu.py

Loading...
View Options

swh/loader/package/loader.py

Loading...
View Options

swh/loader/package/npm.py

Loading...
View Options

swh/loader/package/pypi.py

Loading...
View Options

swh/loader/package/tasks.py

Loading...
View Options

swh/loader/package/tests/common.py

Loading...
View Options

swh/loader/package/tests/conftest.py

Loading...
View Options

swh/loader/package/tests/data/http_deb.debian.org/debian__pool_contrib_c_cicero_cicero_0.7.2-3.diff.gz

Loading...
View Options

swh/loader/package/tests/data/http_deb.debian.org/debian__pool_contrib_c_cicero_cicero_0.7.2-3.dsc

Loading...
View Options

swh/loader/package/tests/data/http_deb.debian.org/debian__pool_contrib_c_cicero_cicero_0.7.2.orig.tar.gz

Loading...
View Options

swh/loader/package/tests/data/http_deb.debian.org/onefile.txt

Loading...
View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/1_private_666_meta

Loading...
View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/1_private_666_raw

Loading...
View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/hello_2.10.json

Loading...
View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/hello_2.10.orig.tar.gz

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.tar.gz

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.zip

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.2.0.zip

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.3.0.zip

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.4.0.zip

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/nexter-1.1.0.tar.gz

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/nexter-1.1.0.zip

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_70_97_c49fb8ec24a7aaab54c3dbfbb5a6ca1431419d9ee0f6c363d9ad01d2b8b1_0805nexter-1.3.0.zip

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip_visit1

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip

Loading...
View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip_visit1

Loading...
View Options

swh/loader/package/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.1.0.tar.gz

Loading...
View Options

swh/loader/package/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.2.0.tar.gz

Loading...
View Options

swh/loader/package/tests/data/https_pypi.org/pypi_0805nexter_json

Loading...
View Options

swh/loader/package/tests/data/https_pypi.org/pypi_0805nexter_json_visit1

Loading...
View Options

swh/loader/package/tests/data/https_pypi.org/pypi_nexter_json

Loading...
View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.2.tgz

Loading...
View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.3.tgz

Loading...
View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.4.tgz

Loading...
View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.5.tgz

Loading...
View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.1.0.tgz

Loading...
View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.2.0.tgz

Loading...
View Options

swh/loader/package/tests/data/https_replicate.npmjs.com/org

Loading...
View Options

swh/loader/package/tests/data/https_replicate.npmjs.com/org_visit1

Loading...
View Options

swh/loader/package/tests/data/loader.yml

Loading...
View Options

swh/loader/package/tests/resources/tarballs/8sync-0.1.0.tar.gz

Loading...
View Options

swh/loader/package/tests/test_common.py

Loading...
View Options

swh/loader/package/tests/test_debian.py

Loading...
View Options

swh/loader/package/tests/test_deposit.py

Loading...
View Options

swh/loader/package/tests/test_gnu.py

Loading...
View Options

swh/loader/package/tests/test_loader.py

Loading...
View Options

swh/loader/package/tests/test_npm.py

Loading...
View Options

swh/loader/package/tests/test_pypi.py

Loading...
View Options

swh/loader/package/tests/test_utils.py

Loading...
View Options

swh/loader/package/utils.py

Loading...
View Options

tox.ini

Loading...
About · Developer information · Legal