Diffusion Generic VCS/Package Loader 6288012791ab

Merge branch 'package-loader'
6288012791ab
Actions

Description

Merge branch 'package-loader'

Related T1389

Details

Provenance

ardumont	Authored on Nov 4 2019, 12:05 PM
ardumont	Pushed on Nov 4 2019, 2:10 PM

Differential Revision

D2214: Land new package loader implementations

Parents

rDLDBASE1c5fc76a41e0: gnu/archive: Merge gnu/archive loader behavior into an archive loader
rDLDBASE8da6f5a9763c: core/loader: Remove origin id references and fix tests

Branches

Unknown

Tags

Unknown

Build Status

Buildable 8797
Build 12821: test-and-build	Jenkins

Event Timeline

ardumont added an edge: D2214: Land new package loader implementations.Nov 4 2019, 2:10 PM

ardumont committed rDLDBASE6288012791ab: Merge branch 'package-loader' (authored by ardumont).Nov 4 2019, 12:05 PM

Harbormaster failed to build B8797: rDLDBASE6288012791ab: Merge branch 'package-loader'!Nov 4 2019, 2:13 PM

Merged Changes

This commit merges a very large number of changes. Only the first 50 are shown.

1c5fc76a41e0 gnu/archive: Merge gnu/archive loader behavior into an archive loader
Author
ardumont
Revisions
D2214: Land new package loader implementations
Browse Repository
Failed
No Audits
4c3646482cc8 gnu: Move version parsing logic to lister
Author
ardumont
Revisions
D2214: Land new package loader implementations
Browse Repository
Failed
No Audits
9cb5ae53cc57 loader.package: Reference tasks for package loaders
Author
ardumont
Revisions
D2214: Land new package loader implementations
Browse Repository
Failed
No Audits
b84fbc58eafc debian.loader: Add tests with one package with multiple versions
Author
ardumont
Browse Repository
No Builds
No Audits
8b08c1f1abcf package.loader: Fix serialization issue with lazy structure
Author
ardumont
Browse Repository
No Builds
No Audits
a549e727cf19 npm: Move helper functions to the bottom of the file
Author
ardumont
Browse Repository
No Builds
No Audits
c5014a184387 pypi: Move helper functions to the bottom of the file
Author
ardumont
Browse Repository
No Builds
No Audits
cbeab5cf0082 loader.package.tests: Use pg storage backend for tests
Author
ardumont
Browse Repository
No Builds
No Audits
2c3bcb62d8a1 loader.package.tests: Make sure we do not request outside services
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
402c3db6bc93 debian: No head alias for debian implementation
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
b60abda87dc3 package.*: Unify release name computation
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
a1da93b43a2e package.loader: Fix snapshot branch computations
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
cfe6be0793ac fix and add debug statement in case the snapshot is not found
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
7179007a81bc package: Manipulate package information instead of artifacts
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
5389c9c7f2b4 package.debian: Check for known revisions using snapshot if any
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
90d9f8861360 debian: Fix edge case about date
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
805863b41edd debian: Explicit we visit twice the same artifacts across 2 visits
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
b2db9931144d debian.loader: Update typing annotations
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
56b2e066cfa6 package.loader: Update comment
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
65ac1a06b315 test_debian: Add test around get_package_metadata function
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
c0e9a8394120 debian.tests: Update pytest_plugin version use
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
65b0a7570b61 debian: Expose debian loader implementation first
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
3310b012e389 debian: Simplify build revision code
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
361e84d127f5 debian: Remove duplicated computations step
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
245259adc2e2 package.debian: Compute the dsc information
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
c19a63428ba9 package.loader: Clarify download_package contract
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
58c532fd62c5 debian: Implement debian loader with package loader mechanism
Author
ardumont
Revisions
D2135: package.debian: Re-implement debian loader within the package loader mechanism
Browse Repository
No Builds
No Audits
0f217d7cc33a Remove unused dataset
Author
ardumont
Browse Repository
No Builds
No Audits
fbf387fa9318 package.tests: Update pytest_plugin version use
Author
ardumont
Browse Repository
No Builds
No Audits
58f2927a148b tox.ini: Install a py3-dev environment to ease development
Author
ardumont
Browse Repository
No Builds
No Audits
89f05060ef37 package: Fix py3.5 incompatibilities
Author
ardumont
Browse Repository
No Builds
No Audits
235164d71d41 tests: Use swh.core's requests_mock_datadir fixture
Author
ardumont
Browse Repository
No Builds
No Audits
1543ec5315eb Use main filtering/buffering storage implementations
Author
ardumont
Browse Repository
No Builds
No Audits
32416802be2b tests: Introduce datadir fixture
Author
ardumont
Browse Repository
No Builds
No Audits
794f75b49364 tests: Allow "local_get" fixture(s) data location override
Author
ardumont
Browse Repository
No Builds
No Audits
44b155cb302b package.gnu: Drop the unused package parameter
Author
ardumont
Browse Repository
No Builds
No Audits
1623e211424b test_pypi: Add multiple visits with no change scenario
Author
ardumont
Browse Repository
No Builds
No Audits
ee81624d73ce test_pypi: Add multiple sdist artifacts per pypi version scenario
Author
ardumont
Browse Repository
No Builds
No Audits
f309320289da package.loader: Improve snapshot creation routine
Author
ardumont
Browse Repository
No Builds
No Audits
9d3c2c9a02bf package.gnu: Add incremental visit behavior for gnu loader
Author
ardumont
Browse Repository
No Builds
No Audits
0ccf3326a992 package.deposit: Update deposit status when successful loading
Author
ardumont
Browse Repository
No Builds
No Audits
25abef596a90 package.deposit: Ingest origin metadata after successful loading
Author
ardumont
Browse Repository
No Builds
No Audits
0ed589a84023 test_deposit: Clean up spurious assertion
Author
ardumont
Browse Repository
No Builds
No Audits
b882f15aba39 package.loader: Make visit_date an instance attribute of loader
Author
ardumont
Browse Repository
No Builds
No Audits
a27eea7227f9 package.loader: Unexpected loading failure returns a failed status
Author
ardumont
Browse Repository
No Builds
No Audits
0c0eab693829 test_deposit: Check revision metadata structure
Author
ardumont
Browse Repository
No Builds
No Audits
5f74caae0f3b package.*.tests: Remove redundant assertion checks
Author
ardumont
Browse Repository
No Builds
No Audits
599fd7396c4b package.deposit: Add basic deposit implementation
Author
ardumont
Browse Repository
No Builds
No Audits
4a27c168bd77 test_utils: Properly check exception message with pytest.raises
Author
ardumont
Browse Repository
No Builds
No Audits
9954fce10ee5 loader.package: Improve metadata normalization implementation
Author
ardumont
Browse Repository
No Builds
No Audits

Changes (72)

Path

Size

MANIFEST.in

requirements-swh.txt

requirements-test.txt

requirements.txt

swh/

loader/

package/

tests/

debian_pool_contrib_c_cicero_cicero_0.7.2-3.diff.gz

debian_pool_contrib_c_cicero_cicero_0.7.2-3.dsc

debian_pool_contrib_c_cicero_cicero_0.7.2-4.diff.gz

debian_pool_contrib_c_cicero_cicero_0.7.2-4.dsc

debian_pool_contrib_c_cicero_cicero_0.7.2.orig.tar.gz

onefile.txt

https_deposit.softwareheritage.org

1_private_666_meta

1_private_666_raw

hello_2.10.json

hello_2.10.orig.tar.gz

https_files.pythonhosted.org

0805nexter-1.1.0.tar.gz

packages_70_97_c49fb8ec24a7aaab54c3dbfbb5a6ca1431419d9ee0f6c363d9ad01d2b8b1_0805nexter-1.3.0.zip

packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip

packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip_visit1

packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip

packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip_visit1

https_ftp.gnu.org

gnu_8sync_8sync-0.2.0.tar.gz

https_pypi.org

pypi_0805nexter_json

pypi_0805nexter_json_visit1

pypi_nexter_json

https_registry.npmjs.org

https_replicate.npmjs.com

org

org_visit1

	data/	https_ftp.gnu.org/
	resources/	tarballs/

	gnu_8sync_8sync-0.1.0.tar.gz
	8sync-0.1.0.tar.gz

rDLDBASE6288012791ab

View Options

MANIFEST.in

View Options

requirements-swh.txt

View Options

requirements-test.txt

View Options

requirements.txt

View Options

swh/loader/package/init.py

View Options

swh/loader/package/archive.py

View Options

swh/loader/package/debian.py

View Options

swh/loader/package/deposit.py

View Options

swh/loader/package/loader.py

View Options

swh/loader/package/npm.py

View Options

swh/loader/package/pypi.py

View Options

swh/loader/package/tasks.py

View Options

swh/loader/package/tests/common.py

View Options

swh/loader/package/tests/conftest.py

View Options

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-3.diff.gz

View Options

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-3.dsc

View Options

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-4.diff.gz

View Options

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-4.dsc

View Options

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2.orig.tar.gz

View Options

swh/loader/package/tests/data/http_deb.debian.org/onefile.txt

View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/1_private_666_meta

View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/1_private_666_raw

View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/hello_2.10.json

View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/hello_2.10.orig.tar.gz

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.tar.gz

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.2.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.3.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.4.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/nexter-1.1.0.tar.gz

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/nexter-1.1.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_70_97_c49fb8ec24a7aaab54c3dbfbb5a6ca1431419d9ee0f6c363d9ad01d2b8b1_0805nexter-1.3.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip_visit1

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip_visit1

View Options

swh/loader/package/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.1.0.tar.gz

View Options

swh/loader/package/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.2.0.tar.gz

View Options

swh/loader/package/tests/data/https_pypi.org/pypi_0805nexter_json

View Options

swh/loader/package/tests/data/https_pypi.org/pypi_0805nexter_json_visit1

View Options

swh/loader/package/tests/data/https_pypi.org/pypi_nexter_json

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.2.tgz

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.3.tgz

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.4.tgz

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.5.tgz

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.1.0.tgz

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.2.0.tgz

View Options

swh/loader/package/tests/data/https_replicate.npmjs.com/org

View Options

swh/loader/package/tests/data/https_replicate.npmjs.com/org_visit1

View Options

swh/loader/package/tests/resources

View Options

swh/loader/package/tests/resources/tarballs

View Options

swh/loader/package/tests/resources/tarballs/8sync-0.1.0.tar.gz

View Options

swh/loader/package/tests/test_archive.py

View Options

swh/loader/package/tests/test_common.py

View Options

swh/loader/package/tests/test_conftest.py

View Options

swh/loader/package/tests/test_debian.py

View Options

swh/loader/package/tests/test_deposit.py

View Options

swh/loader/package/tests/test_loader.py

View Options

swh/loader/package/tests/test_npm.py

View Options

swh/loader/package/tests/test_pypi.py

View Options

swh/loader/package/tests/test_tasks.py

View Options

swh/loader/package/tests/test_utils.py

View Options

swh/loader/package/utils.py

View Options

Merge branch 'package-loader'6288012791abActions

Description

Details

Event Timeline

Merged Changes

Changes (72)

rDLDBASE6288012791ab

MANIFEST.in

requirements-swh.txt

requirements-test.txt

requirements.txt

swh/loader/package/__init__.py

swh/loader/package/archive.py

swh/loader/package/debian.py

swh/loader/package/deposit.py

swh/loader/package/loader.py

swh/loader/package/npm.py

swh/loader/package/pypi.py

swh/loader/package/tasks.py

swh/loader/package/tests/common.py

swh/loader/package/tests/conftest.py

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-3.diff.gz

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-3.dsc

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-4.diff.gz

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-4.dsc

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2.orig.tar.gz

swh/loader/package/tests/data/http_deb.debian.org/onefile.txt

swh/loader/package/tests/data/https_deposit.softwareheritage.org/1_private_666_meta

swh/loader/package/tests/data/https_deposit.softwareheritage.org/1_private_666_raw

swh/loader/package/tests/data/https_deposit.softwareheritage.org/hello_2.10.json

swh/loader/package/tests/data/https_deposit.softwareheritage.org/hello_2.10.orig.tar.gz

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.tar.gz

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.2.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.3.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.4.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/nexter-1.1.0.tar.gz

swh/loader/package/tests/data/https_files.pythonhosted.org/nexter-1.1.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_70_97_c49fb8ec24a7aaab54c3dbfbb5a6ca1431419d9ee0f6c363d9ad01d2b8b1_0805nexter-1.3.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip_visit1

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip_visit1

swh/loader/package/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.1.0.tar.gz

swh/loader/package/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.2.0.tar.gz

swh/loader/package/tests/data/https_pypi.org/pypi_0805nexter_json

swh/loader/package/tests/data/https_pypi.org/pypi_0805nexter_json_visit1

swh/loader/package/tests/data/https_pypi.org/pypi_nexter_json

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.2.tgz

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.3.tgz

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.4.tgz

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.5.tgz

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.1.0.tgz

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.2.0.tgz

swh/loader/package/tests/data/https_replicate.npmjs.com/org

swh/loader/package/tests/data/https_replicate.npmjs.com/org_visit1

swh/loader/package/tests/resources

swh/loader/package/tests/resources/tarballs

swh/loader/package/tests/resources/tarballs/8sync-0.1.0.tar.gz

swh/loader/package/tests/test_archive.py

swh/loader/package/tests/test_common.py

swh/loader/package/tests/test_conftest.py

swh/loader/package/tests/test_debian.py

swh/loader/package/tests/test_deposit.py

swh/loader/package/tests/test_loader.py

swh/loader/package/tests/test_npm.py

swh/loader/package/tests/test_pypi.py

swh/loader/package/tests/test_tasks.py

swh/loader/package/tests/test_utils.py

swh/loader/package/utils.py

tox.ini

Merge branch 'package-loader'
6288012791ab
Actions

swh/loader/package/init.py