Land new package loader implementations
ClosedPublic
Actions

Authored by ardumont on Nov 4 2019, 2:04 PM.

Details

Reviewers

None

Group Reviewers

Commits

rDLDBASE918a720673d4: mypy: Fix type annotations
rDLDBASE6288012791ab: Merge branch 'package-loader'
rDLDBASE1c5fc76a41e0: gnu/archive: Merge gnu/archive loader behavior into an archive loader
rDLDBASE4c3646482cc8: gnu: Move version parsing logic to lister
rDLDBASE9cb5ae53cc57: loader.package: Reference tasks for package loaders

Summary

I do not expect review.
This is to ensure the ci is fine.

This is the work we did (@douardda and me) on the package loader
re-implementations (debian, deposit, npm, pypi, archive/gnu).

It remains some checks on the debian loader to do but that should not prevent
the merge. That will simplify the future diffs for it.

We decided to merge it with the master branch because:

master diverged (mypy got introduced, fixes: scheduler tasks, storage migration)
storage migration in between broke the package-loader branch for unrelated reasons (fixed in master branch)
too many commits which would make the ci crumble if sequential builds are done
in the end, it's been opened for too long now

Related T1389 T2021 T2022 T2023 T2024 T2025

Test Plan

tox

Diff Detail

Repository

rDLDBASE Generic VCS/Package Loader

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 8793
Build 12817: tox-on-jenkins	Jenkins
Build 12816: arc lint + arc unit

Event Timeline

ardumont created this revision.Nov 4 2019, 2:04 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptNov 4 2019, 2:04 PM

ardumont mentioned this in D2164: loader.package: Reference tasks for package loaders.Nov 4 2019, 2:04 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/225/ for more details.

Harbormaster completed remote builds in B8792: Diff 7627.Nov 4 2019, 2:06 PM

Plug to master branch

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/226/ for more details.

Harbormaster completed remote builds in B8793: Diff 7628.Nov 4 2019, 2:08 PM

This revision was not accepted when it landed; it landed in state Needs Review.Nov 4 2019, 2:10 PM

Closed by commit rDLDBASE9cb5ae53cc57: loader.package: Reference tasks for package loaders (authored by ardumont). · Explain Why

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDLDBASE9cb5ae53cc57: loader.package: Reference tasks for package loaders.

ardumont added a commit: rDLDBASE4c3646482cc8: gnu: Move version parsing logic to lister.

ardumont added a commit: rDLDBASE1c5fc76a41e0: gnu/archive: Merge gnu/archive loader behavior into an archive loader.

ardumont added a commit: rDLDBASE6288012791ab: Merge branch 'package-loader'.

ardumont added a commit: rDLDBASE918a720673d4: mypy: Fix type annotations.

Revision Contents
Changeset List

Path

Size

MANIFEST.in

1 line

mypy.ini

17 lines

requirements-swh.txt

4 lines

requirements-test.txt

2 lines

requirements.txt

3 lines

swh/

loader/

package/

19 lines

136 lines

394 lines

154 lines

695 lines

296 lines

197 lines

37 lines

tests/

common.py

133 lines

conftest.py

51 lines

data/

http_deb.debian.org/

debian_pool_contrib_c_cicero_cicero_0.7.2-3.diff.gz

debian_pool_contrib_c_cicero_cicero_0.7.2-3.dsc

44 lines

debian_pool_contrib_c_cicero_cicero_0.7.2-4.diff.gz

debian_pool_contrib_c_cicero_cicero_0.7.2-4.dsc

43 lines

debian_pool_contrib_c_cicero_cicero_0.7.2.orig.tar.gz

onefile.txt

1 line

https_deposit.softwareheritage.org/

1_private_666_meta

1 line

1_private_666_raw

1 line

hello_2.10.json

80 lines

hello_2.10.orig.tar.gz

https_files.pythonhosted.org/

0805nexter-1.1.0.tar.gz

1 line

1 line

packages_70_97_c49fb8ec24a7aaab54c3dbfbb5a6ca1431419d9ee0f6c363d9ad01d2b8b1_0805nexter-1.3.0.zip

1 line

packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip

1 line

packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip_visit1

1 line

packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip

1 line

packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip_visit1

1 line

https_ftp.gnu.org/

gnu_8sync_8sync-0.2.0.tar.gz

https_pypi.org/

pypi_0805nexter_json

95 lines

pypi_0805nexter_json_visit1

114 lines

pypi_nexter_json

74 lines

https_registry.npmjs.org/

https_replicate.npmjs.com/

org

191 lines

org_visit1

347 lines

	data/	https_ftp.gnu.org/
	resources/	tarballs/

	gnu_8sync_8sync-0.1.0.tar.gz
	8sync-0.1.0.tar.gz

resources/

tarballs/

344 lines

175 lines

12 lines

371 lines

204 lines

526 lines

661 lines

83 lines

153 lines

111 lines

11 lines

Commit	Tree	Parents	Author	Summary	Date
918a720673d4	4fa8aa075f4c	6288012791ab	Antoine R. Dumont (@ardumont)	mypy: Fix type annotations	Nov 4 2019, 1:48 PM
6288012791ab	1c5498c72d09	8da6f5a9763c 1c5fc76a41e0	Antoine R. Dumont (@ardumont)	Merge branch 'package-loader' (Show More…)	Nov 4 2019, 12:05 PM
1c5fc76a41e0	9af2725428d1	4c3646482cc8	Antoine R. Dumont (@ardumont)	gnu/archive: Merge gnu/archive loader behavior into an archive loader (Show More…)	Oct 17 2019, 4:18 AM
4c3646482cc8	7aa84a80dc12	9cb5ae53cc57	Antoine R. Dumont (@ardumont)	gnu: Move version parsing logic to lister (Show More…)	Oct 16 2019, 2:52 PM
9cb5ae53cc57	8bf0ed97bfe2	b84fbc58eafc	Antoine R. Dumont (@ardumont)	loader.package: Reference tasks for package loaders	Oct 18 2019, 4:13 AM
b84fbc58eafc	3a6a6db4f2bd	8b08c1f1abcf	Antoine R. Dumont (@ardumont)	debian.loader: Add tests with one package with multiple versions	Oct 18 2019, 2:32 PM
8b08c1f1abcf	c9497563ae78	a549e727cf19	Antoine R. Dumont (@ardumont)	package.loader: Fix serialization issue with lazy structure	Oct 18 2019, 12:19 PM
a549e727cf19	43b61c26f194	c5014a184387	Antoine R. Dumont (@ardumont)	npm: Move helper functions to the bottom of the file	Oct 18 2019, 3:41 AM
c5014a184387	929e285b2e0f	cbeab5cf0082	Antoine R. Dumont (@ardumont)	pypi: Move helper functions to the bottom of the file	Oct 18 2019, 3:41 AM
cbeab5cf0082	73dbd278a072	2c3bcb62d8a1	Antoine R. Dumont (@ardumont)	loader.package.tests: Use pg storage backend for tests (Show More…)	Oct 17 2019, 3:42 PM
2c3bcb62d8a1	2f50c83a38a9	402c3db6bc93	Antoine R. Dumont (@ardumont)	loader.package.tests: Make sure we do not request outside services	Oct 17 2019, 11:38 AM
402c3db6bc93	e61a48c89111	b60abda87dc3	Antoine R. Dumont (@ardumont)	debian: No head alias for debian implementation	Oct 17 2019, 11:22 AM
b60abda87dc3	c4471ca1ee71	a1da93b43a2e	Antoine R. Dumont (@ardumont)	package.*: Unify release name computation	Oct 17 2019, 5:02 AM
a1da93b43a2e	daf39f46c5a3	cfe6be0793ac	Antoine R. Dumont (@ardumont)	package.loader: Fix snapshot branch computations (Show More…)	Oct 17 2019, 5:01 AM
cfe6be0793ac	6c49d54fbc73	7179007a81bc	Antoine R. Dumont (@ardumont)	fix and add debug statement in case the snapshot is not found	Oct 16 2019, 7:12 PM
7179007a81bc	c938c7c51bb9	5389c9c7f2b4	Antoine R. Dumont (@ardumont)	package: Manipulate package information instead of artifacts	Oct 16 2019, 5:27 PM
5389c9c7f2b4	f51cbfda664f	90d9f8861360	Antoine R. Dumont (@ardumont)	package.debian: Check for known revisions using snapshot if any (Show More…)	Oct 15 2019, 1:56 PM
90d9f8861360	eac16957de81	805863b41edd	Antoine R. Dumont (@ardumont)	debian: Fix edge case about date	Oct 15 2019, 5:06 PM
805863b41edd	59adfb3ebc0c	b2db9931144d	Antoine R. Dumont (@ardumont)	debian: Explicit we visit twice the same artifacts across 2 visits (Show More…)	Oct 15 2019, 1:29 PM
b2db9931144d	edf7eddcc508	56b2e066cfa6	Antoine R. Dumont (@ardumont)	debian.loader: Update typing annotations (Show More…)	Oct 15 2019, 1:14 PM
56b2e066cfa6	f1c0fdfea94b	65ac1a06b315	Antoine R. Dumont (@ardumont)	package.loader: Update comment	Oct 15 2019, 12:16 PM
65ac1a06b315	b1100f6eaca1	c0e9a8394120	Antoine R. Dumont (@ardumont)	test_debian: Add test around get_package_metadata function	Oct 15 2019, 12:15 PM
c0e9a8394120	57722bd7e03e	65b0a7570b61	Antoine R. Dumont (@ardumont)	debian.tests: Update pytest_plugin version use	Oct 14 2019, 6:09 PM
65b0a7570b61	c85c9964c9bb	3310b012e389	Antoine R. Dumont (@ardumont)	debian: Expose debian loader implementation first (Show More…)	Oct 14 2019, 2:24 PM
3310b012e389	05ae0f747545	361e84d127f5	Antoine R. Dumont (@ardumont)	debian: Simplify build revision code	Oct 14 2019, 2:03 PM
361e84d127f5	e662ac11afd4	245259adc2e2	Antoine R. Dumont (@ardumont)	debian: Remove duplicated computations step (Show More…)	Oct 14 2019, 1:51 PM
245259adc2e2	e0ba77ce4749	c19a63428ba9	Antoine R. Dumont (@ardumont)	package.debian: Compute the dsc information (Show More…)	Oct 14 2019, 1:48 PM
c19a63428ba9	dee13612c37e	58c532fd62c5	Antoine R. Dumont (@ardumont)	package.loader: Clarify download_package contract (Show More…)	Oct 14 2019, 12:28 PM
58c532fd62c5	b90fb0ba27f7	0f217d7cc33a	Antoine R. Dumont (@ardumont)	debian: Implement debian loader with package loader mechanism (Show More…)	Oct 12 2019, 7:50 PM
0f217d7cc33a	e391bdfcdf1c	fbf387fa9318	Antoine R. Dumont (@ardumont)	Remove unused dataset	Oct 14 2019, 6:08 PM
fbf387fa9318	96a6d1802b44	58f2927a148b	Antoine R. Dumont (@ardumont)	package.tests: Update pytest_plugin version use	Oct 14 2019, 6:02 PM
58f2927a148b	f0b2b4c857d7	89f05060ef37	Antoine R. Dumont (@ardumont)	tox.ini: Install a py3-dev environment to ease development (Show More…)	Oct 12 2019, 7:46 PM
89f05060ef37	98bc81fced48	235164d71d41	Antoine R. Dumont (@ardumont)	package: Fix py3.5 incompatibilities (Show More…)	Oct 14 2019, 9:47 AM
235164d71d41	81a53bbe0b04	1543ec5315eb	Antoine R. Dumont (@ardumont)	tests: Use swh.core's requests_mock_datadir fixture	Oct 9 2019, 11:27 AM
1543ec5315eb	5c1446089f44	32416802be2b	Antoine R. Dumont (@ardumont)	Use main filtering/buffering storage implementations (Show More…)	Oct 8 2019, 5:36 PM
32416802be2b	6835fcbd487f	794f75b49364	Antoine R. Dumont (@ardumont)	tests: Introduce datadir fixture (Show More…)	Oct 7 2019, 11:53 AM
794f75b49364	f6faf507c165	44b155cb302b	Antoine R. Dumont (@ardumont)	tests: Allow "local_get" fixture(s) data location override	Oct 7 2019, 11:26 AM
44b155cb302b	1ca6dfd5893e	1623e211424b	Antoine R. Dumont (@ardumont)	package.gnu: Drop the unused package parameter (Show More…)	Oct 5 2019, 6:10 PM
1623e211424b	df16fb5849c2	ee81624d73ce	Antoine R. Dumont (@ardumont)	test_pypi: Add multiple visits with no change scenario	Oct 4 2019, 1:23 PM
ee81624d73ce	c1e77d06163f	f309320289da	Antoine R. Dumont (@ardumont)	test_pypi: Add multiple sdist artifacts per pypi version scenario	Oct 4 2019, 12:31 PM
f309320289da	788191dff821	9d3c2c9a02bf	Antoine R. Dumont (@ardumont)	package.loader: Improve snapshot creation routine	Oct 4 2019, 12:14 PM
9d3c2c9a02bf	ebb48b6c762c	0ccf3326a992	Antoine R. Dumont (@ardumont)	package.gnu: Add incremental visit behavior for gnu loader	Oct 4 2019, 11:30 AM
0ccf3326a992	fec7def6021f	25abef596a90	Antoine R. Dumont (@ardumont)	package.deposit: Update deposit status when successful loading	Oct 3 2019, 4:52 PM
25abef596a90	fee93f1aa6bf	0ed589a84023	Antoine R. Dumont (@ardumont)	package.deposit: Ingest origin metadata after successful loading	Oct 3 2019, 4:11 PM
0ed589a84023	5b3a4f61f013	b882f15aba39	Antoine R. Dumont (@ardumont)	test_deposit: Clean up spurious assertion	Oct 3 2019, 4:11 PM
b882f15aba39	6d0c609ef480	a27eea7227f9	Antoine R. Dumont (@ardumont)	package.loader: Make visit_date an instance attribute of loader	Oct 3 2019, 3:44 PM
a27eea7227f9	8668c1aa6e9e	0c0eab693829	Antoine R. Dumont (@ardumont)	package.loader: Unexpected loading failure returns a failed status	Oct 3 2019, 3:07 PM
0c0eab693829	72d3ed5c2638	5f74caae0f3b	Antoine R. Dumont (@ardumont)	test_deposit: Check revision metadata structure	Oct 3 2019, 2:42 PM
5f74caae0f3b	f54f3a89c5c6	599fd7396c4b	Antoine R. Dumont (@ardumont)	package.*.tests: Remove redundant assertion checks	Oct 3 2019, 2:39 PM
599fd7396c4b	ef559762e356	4a27c168bd77	Antoine R. Dumont (@ardumont)	package.deposit: Add basic deposit implementation (Show More…)	Oct 2 2019, 4:22 PM
4a27c168bd77	777fc383e7bb	9954fce10ee5	Antoine R. Dumont (@ardumont)	test_utils: Properly check exception message with pytest.raises	Oct 2 2019, 1:47 PM
9954fce10ee5	64d8bf215e82	c2c163bea9bd	Antoine R. Dumont (@ardumont)	loader.package: Improve metadata normalization implementation (Show More…)	Oct 2 2019, 1:45 PM
c2c163bea9bd	fcf75ac30837	98f6deb305bd	Antoine R. Dumont (@ardumont)	package.tests.pypi: Use metadata instead of sdist notion	Oct 1 2019, 7:27 PM
98f6deb305bd	872f200d119c	7551d82b8996	Antoine R. Dumont (@ardumont)	package.loader: Normalize revision metadata (Show More…)	Oct 1 2019, 3:19 PM
7551d82b8996	53bd395d3b5f	9788991c3e1d	Antoine R. Dumont (@ardumont)	package.loader: Remove artifact retrieval indirection	Oct 1 2019, 3:19 PM
9788991c3e1d	079d3b77c4d7	1d3d1dece1da	Antoine R. Dumont (@ardumont)	package.loader: Remove todo	Sep 30 2019, 3:55 PM
1d3d1dece1da	ad37f8d89160	80b9fc599ed3	Antoine R. Dumont (@ardumont)	test_pypi: Check the last origin visit is ok	Sep 30 2019, 3:53 PM
80b9fc599ed3	c4403af0f972	b1478249ee15	Antoine R. Dumont (@ardumont)	package.npm: Add incremental visit behavior for npm loader	Sep 30 2019, 3:52 PM
b1478249ee15	89c81b516324	7b95ef08eb70	Antoine R. Dumont (@ardumont)	package.npm: Add new loading visit test scenario	Sep 30 2019, 3:39 PM
7b95ef08eb70	e089795acd15	81242cd30ee5	Antoine R. Dumont (@ardumont)	package.loader: Add incremental visit behavior for pypi loader	Sep 30 2019, 3:21 PM
81242cd30ee5	ff0cb2321f35	fcfdf480e3c8	Antoine R. Dumont (@ardumont)	package.loader: Remove package loader's missing behavior (Show More…)	Sep 29 2019, 4:35 PM
fcfdf480e3c8	6661a805cb80	5f3c3608adc6	Antoine R. Dumont (@ardumont)	package.utils: Add back download hashes check behavior when provided	Sep 29 2019, 4:30 PM
5f3c3608adc6	c173b936419d	b5fdd598e642	Antoine R. Dumont (@ardumont)	package.tests: Add coverage to common test utilities	Sep 27 2019, 4:38 PM
b5fdd598e642	52ddb7296778	e0adf6f72444	Antoine R. Dumont (@ardumont)	storages: Add tests around proxy storages	Sep 27 2019, 3:34 PM
e0adf6f72444	6ba1cc6aa197	4ae274293b3c	Antoine R. Dumont (@ardumont)	FilteringProxyStorage: Move getattr method to its rightful position	Sep 27 2019, 3:01 PM
4ae274293b3c	40e6b5ffb26f	5b0fbd279fc1	Antoine R. Dumont (@ardumont)	storage.BufferingStorageProxy: Allow buffering storage configuration	Sep 27 2019, 3:00 PM
5b0fbd279fc1	9ecd490bbd69	2a94112c14f5	Antoine R. Dumont (@ardumont)	storage: Add Buffering proxy storage implementation	Sep 27 2019, 2:42 PM
2a94112c14f5	2b40451156f9	aa726d1c0163	Antoine R. Dumont (@ardumont)	package.storage: Move internal helper methods	Sep 27 2019, 1:14 PM
aa726d1c0163	4cafb5aa2cea	211f8c83d97b	Antoine R. Dumont (@ardumont)	loader.package: Provide a default implementation to fetch artifact (Show More…)	Sep 27 2019, 10:32 AM
211f8c83d97b	878c83a03348	b7de277d8283	Antoine R. Dumont (@ardumont)	package.npm: Reuse same api info client for both pypi/npm	Sep 27 2019, 10:27 AM
b7de277d8283	1dc04140f991	e8777c501db7	Antoine R. Dumont (@ardumont)	package.pypi: Rename sdist_parse to extract_intrinsic_metadata	Sep 27 2019, 10:13 AM
e8777c501db7	ee82797e6433	002edabb9d99	Antoine R. Dumont (@ardumont)	package.storage: Use the same typed cache everywhere	Sep 26 2019, 5:43 PM
002edabb9d99	39f7352cdd3c	9e3c0dc0e5f6	Antoine R. Dumont (@ardumont)	package.loader: Improve proxy storage's filtering implementation	Sep 26 2019, 4:07 PM
9e3c0dc0e5f6	459e62027a11	0584c3691907	Antoine R. Dumont (@ardumont)	package.storage: Avoid redefining passthrough methods	Sep 26 2019, 4:06 PM
0584c3691907	95481367acff	c540a1bd29c5	Antoine R. Dumont (@ardumont)	package.loader: Add a fixme about revision's metadata keys	Sep 26 2019, 4:05 PM
c540a1bd29c5	1f118e75ea24	7ae53a0577c4	Antoine R. Dumont (@ardumont)	package.pypi: Use correct condition to check for PKG-INFO search	Sep 26 2019, 4:03 PM
7ae53a0577c4	8f62098136ed	0ea3078f0bc3	Antoine R. Dumont (@ardumont)	package.tests.conftest: Align regexp	Sep 26 2019, 4:03 PM
0ea3078f0bc3	d0ebd90faa30	af3475f4bb63	Antoine R. Dumont (@ardumont)	loader.packages: Fix various typos	Sep 26 2019, 4:02 PM
af3475f4bb63	17a51cb07201	dce2ca5c00f4	Antoine R. Dumont (@ardumont)	package.loader: Actually set an empty snapshot if no branch	Sep 26 2019, 4:02 PM
dce2ca5c00f4	ea446909ac4a	a9783997aa84	Antoine R. Dumont (@ardumont)	package.gnu: Actually sort input tarballs	Sep 26 2019, 4:01 PM
a9783997aa84	ee885b184d13	763df089d4d2	Antoine R. Dumont (@ardumont)	package.loader: Remove dealt with FIXME	Sep 26 2019, 4:01 PM
763df089d4d2	f6e8a2264125	c679e54d30f5	Antoine R. Dumont (@ardumont)	package.loader: Update remaining action items	Sep 26 2019, 3:59 PM
c679e54d30f5	6c3d82f0f52b	57ab9a506f02	Antoine R. Dumont (@ardumont)	ProxyStorage: Refactor common filtering behavior	Sep 26 2019, 1:14 PM
57ab9a506f02	ed63299435af	c1cd90cb1cd0	Antoine R. Dumont (@ardumont)	ProxyStorage: Add filtering routine before sending data to storage	Sep 26 2019, 11:37 AM
c1cd90cb1cd0	cdbed54a4fa3	b2338fba0cee	Antoine R. Dumont (@ardumont)	loader.package: Add a first proxy storage draft (Show More…)	Sep 25 2019, 4:22 PM
b2338fba0cee	3ffcdb127b50	a5ac8f4acfb0	Antoine R. Dumont (@ardumont)	test: Loading pypi origin with new artifact results in new snapshot	Sep 25 2019, 3:50 PM
a5ac8f4acfb0	684b1f49ffd6	50fbc8447838	Antoine R. Dumont (@ardumont)	conftest: Add better support for multiple visits	Sep 25 2019, 3:01 PM
50fbc8447838	18059a4736ba	0c096a710ce3	Antoine R. Dumont (@ardumont)	conftest: Add coverage to local_get fixture factory	Sep 25 2019, 2:23 PM
0c096a710ce3	c06b85e9a36e	575870b9745b	Antoine R. Dumont (@ardumont)	conftest: Allow multiple visits with different responses (Show More…)	Sep 25 2019, 2:14 PM
575870b9745b	9351f2bd1708	cc518575bd62	Antoine R. Dumont (@ardumont)	package.loader: Update origin visit status in all cases	Sep 25 2019, 1:37 PM
cc518575bd62	3925204ad7e7	939c0f54da7c	Antoine R. Dumont (@ardumont)	tests/pypi: Add loading origin with no release artifact scenario	Sep 25 2019, 12:18 PM
939c0f54da7c	54f1779cc1b5	f382f2778db2	Antoine R. Dumont (@ardumont)	pypi: Add support for loading repositories with missing artifacts	Sep 25 2019, 12:02 PM
f382f2778db2	3eb407cfb4f5	c86c9f2b17a9	Antoine R. Dumont (@ardumont)	conftest: Allow local_get fixture to ignore specific urls (Show More…)	Sep 25 2019, 11:30 AM
c86c9f2b17a9	6a70e03d0ea3	fe74225f34c7	Antoine R. Dumont (@ardumont)	conftest: Move fixture related functions to conftest	Sep 25 2019, 11:22 AM
fe74225f34c7	461204c233af	829345d0b11e	Antoine R. Dumont (@ardumont)	test_pypi: Remove unused comments	Sep 25 2019, 11:04 AM
829345d0b11e	d1c05a00faaa	3a964bb4179d	David Douard	package.tests: improve get_response_cb docstring and add a bit of logging	Sep 25 2019, 10:21 AM
3a964bb4179d	d48ab9830ce5	7521f01a990e	David Douard	package.tests: make get_response_cb generate a 404 if the requested file does… (Show More…)	Sep 25 2019, 10:18 AM
7521f01a990e	9e4461b54db8	49e49843bc47	David Douard	package.tests: add a local_get fixture that install the get_response_cb mock (Show More…)	Sep 25 2019, 10:15 AM
49e49843bc47	5213b5486959	1e90f7c5df2b	Antoine R. Dumont (@ardumont)	test_npm: Drop unittest tests format	Sep 24 2019, 5:03 PM
1e90f7c5df2b	c71bb5379e99	947991d497ae	Antoine R. Dumont (@ardumont)	MANIFEST.in: Add the tests resources in package	Sep 24 2019, 4:38 PM
947991d497ae	eb4bee29ead9	b1935a9a8556	Antoine R. Dumont (@ardumont)	tests: Remove tox's specific environment setup (Show More…)	Sep 24 2019, 4:37 PM
b1935a9a8556	e4ae91ec9e98	0d63412299b4	David Douard	package.tests: use a fixture to configure the loaders (Show More…)	Sep 24 2019, 4:23 PM
0d63412299b4	7ab64df10e47	fec1fe1f5d90	Antoine R. Dumont (@ardumont)	test_gnu: Add missing space	Sep 24 2019, 2:36 PM
fec1fe1f5d90	a7f5cd777c5d	97ab4e3db3b9	Antoine R. Dumont (@ardumont)	package.npm: Add basic npm loader implementation	Sep 24 2019, 1:54 PM
97ab4e3db3b9	40f82a9037bc	0e1be5e11016	Antoine R. Dumont (@ardumont)	package.loader: Catch more general exceptions	Sep 24 2019, 1:54 PM
0e1be5e11016	b5d9f8f6c5a9	76c93a66371b	Antoine R. Dumont (@ardumont)	package.loader: Add logging debug statements	Sep 24 2019, 1:54 PM
76c93a66371b	7cbe5e78cd12	b316d36f5d48	Antoine R. Dumont (@ardumont)	package.loaders: Fix type checking warnings	Sep 23 2019, 6:41 PM
b316d36f5d48	20175f8b52c1	664b29a2a9da	Antoine R. Dumont (@ardumont)	loader: Deal with loading error	Sep 23 2019, 6:24 PM
664b29a2a9da	5e316f4e2e07	926c3171af64	Antoine R. Dumont (@ardumont)	gnu: Improve yet again the version matching regexp	Sep 23 2019, 4:26 PM
926c3171af64	8245968b8b6b	a4177221972d	Antoine R. Dumont (@ardumont)	gnu: Improve slightly release number extraction	Sep 23 2019, 3:45 PM
a4177221972d	b62c5e9d1ce5	735c5e62afe3	Antoine R. Dumont (@ardumont)	gnu: Refactor using PackageLoader as basis	Sep 23 2019, 1:32 PM
735c5e62afe3	836e71bf9f53	c42079e5063e	Antoine R. Dumont (@ardumont)	package.pypi: Extract download function into its own utils module	Sep 23 2019, 10:22 AM
c42079e5063e	87a6e688ba40	43e88116be4a	Antoine R. Dumont (@ardumont)	package.loader: Fix typo	Sep 23 2019, 10:19 AM
43e88116be4a	c16c5eb93441	3e6bd5630580	Antoine R. Dumont (@ardumont)	package.loader.load: Add missing type annotation	Sep 23 2019, 10:05 AM
3e6bd5630580	c4fb897bd41a	10b09d0b6b05	Antoine R. Dumont (@ardumont)	package.loader: Transform FIXME into a comment	Sep 23 2019, 10:05 AM
10b09d0b6b05	d6c48b38b7ce	89a087201074	Antoine R. Dumont (@ardumont)	package.loader: Add notes about what's not yet implemented	Sep 23 2019, 10:05 AM
89a087201074	df5b419089e7	acde74c2bd08	Antoine R. Dumont (@ardumont)	package.pypi: Make PyPIClient a function (Show More…)	Sep 22 2019, 9:31 AM
acde74c2bd08	52d7b4bd2f99	ba245e681952	Antoine R. Dumont (@ardumont)	test_pypi: Fix pep8 violation	Sep 22 2019, 9:30 AM
ba245e681952	afee9a7c26f9	4a437bb48b7a	Antoine R. Dumont (@ardumont)	package.pypi: Add hash checks	Sep 20 2019, 4:32 PM
4a437bb48b7a	769d8deaa8ce	098005605a37	Antoine R. Dumont (@ardumont)	package.pypi: Clean up dead code	Sep 20 2019, 4:03 PM
098005605a37	93fb223050e9	4086cab9fe6a	Antoine R. Dumont (@ardumont)	package.pypi: Add one loading use case and make the loader work (Show More…)	Sep 20 2019, 4:01 PM
4086cab9fe6a	07252c02e874	2df3bd9e1c85	Antoine R. Dumont (@ardumont)	package.pypi: Add tests around download function	Sep 20 2019, 11:55 AM
2df3bd9e1c85	aa6f8b366394	ac5f1a878a16	Antoine R. Dumont (@ardumont)	loader.pypi: Add tests around sdist_parse	Sep 20 2019, 7:40 AM
ac5f1a878a16	6368ceea413e	b2752142cc04	Antoine R. Dumont (@ardumont)	loader.pypi: Add test around sdist_parse	Sep 20 2019, 7:32 AM
b2752142cc04	c7dd0b4f52b6	902845193523	Antoine R. Dumont (@ardumont)	pypi: Fix pep8 violations	Sep 20 2019, 7:30 AM
902845193523	8c3ce88545ed	26190a555ad8	Antoine R. Dumont (@ardumont)	package.pypi: Add tests around PyPIClient	Sep 20 2019, 12:05 AM
26190a555ad8	3b8e4f75fc3a	f4c44e49bb3e	Antoine R. Dumont (@ardumont)	package.pypi: Update docstrings and type annotations	Sep 19 2019, 10:40 PM
f4c44e49bb3e	3b206586fe0d	03f5a1f90ea9	Antoine R. Dumont (@ardumont)	package.pypi: Add type annotation	Sep 19 2019, 9:44 PM
03f5a1f90ea9	ed048971350f	25aac0ea9f59	Antoine R. Dumont (@ardumont)	test_pypi: Add tests around `author` function	Sep 19 2019, 9:39 PM
25aac0ea9f59	e80404fe6309	64ba90a75197	Antoine R. Dumont (@ardumont)	loader.package: Bootstrap tests	Sep 19 2019, 9:33 PM
64ba90a75197	0bbb7445fdd3	102bc7b38c34	Antoine R. Dumont (@ardumont)	loader.package: Fix pep8 violations	Sep 19 2019, 9:33 PM
102bc7b38c34	1ff26dbefc55	9a456a652ac7	Antoine R. Dumont (@ardumont)	package.pypi: Fix __init__ call	Sep 19 2019, 9:32 PM
9a456a652ac7	7b1f2cf7e952	a62355d56656	Antoine R. Dumont (@ardumont)	package.loader: Fix import	Sep 19 2019, 9:19 PM
a62355d56656	32342ae372e3	3f3e774c47bb	Antoine R. Dumont (@ardumont)	package.loader: Remove spurious fixme	Sep 19 2019, 9:19 PM
3f3e774c47bb	bcdb48b19a0c	fc977e6e92cb	Antoine R. Dumont (@ardumont)	requirements: Add missing pkginfo	Sep 19 2019, 9:18 PM
fc977e6e92cb	fd9f9c9f04c8	a2f2d0f88d40	Antoine R. Dumont (@ardumont)	wip: Add pypi implementation	Sep 19 2019, 4:39 PM
a2f2d0f88d40	2f4607e89e91	8d46a9613603	Antoine R. Dumont (@ardumont)	package.loader: Implement further pypi loader	Sep 18 2019, 6:21 PM
8d46a9613603	3eb2b1db9564	e42a7c6519c1	Antoine R. Dumont (@ardumont)	package.loader: clarify method name intent	Sep 18 2019, 6:21 PM
e42a7c6519c1	366bbfb820e8	99fd2584ea4a	Antoine R. Dumont (@ardumont)	package.loader: Fix method name	Sep 18 2019, 6:21 PM
99fd2584ea4a	59d946225e4c	37b4d313f580	Antoine R. Dumont (@ardumont)	loader.package: Add basic pypi collaborators	Sep 18 2019, 4:28 PM
37b4d313f580	d325c3b04e44	2ad84f608037	Antoine R. Dumont (@ardumont)	package.loader: Simplify message computation	Sep 18 2019, 3:18 PM
2ad84f608037	0a10135a38cd	51d742eb278c	Antoine R. Dumont (@ardumont)	package.loader: Check only configuration	Sep 18 2019, 3:18 PM
51d742eb278c	13f7839baba6	2429a0aac752	Antoine R. Dumont (@ardumont)	package.loader: Add FIXME around exceptions handling	Sep 18 2019, 2:50 PM
2429a0aac752	9f851d6de882	f4d44cd9c45c	Antoine R. Dumont (@ardumont)	package.loader: Improve variable name (Show More…)	Sep 18 2019, 2:43 PM
f4d44cd9c45c	9ac0c9b1d710	271473fb553d	Antoine R. Dumont (@ardumont)	package.loader: No parent revision by default	Sep 18 2019, 2:41 PM
271473fb553d	0f9a48b5e0a1	e1a74f7171d6	Antoine R. Dumont (@ardumont)	loader.package: Deal with origin and visit policy	Sep 18 2019, 2:34 PM
e1a74f7171d6	06d200ff2f94	e3387de22d16	Antoine R. Dumont (@ardumont)	loader.package: Add docstrings to methods	Sep 18 2019, 12:28 PM
e3387de22d16	08a3200452d7	839bc2d44e7d	Antoine R. Dumont (@ardumont)	package.loader: Add FIXME about origin/visit creation/update	Sep 18 2019, 12:28 PM
839bc2d44e7d	83f1437ed1ed	2efaa062ca47	Antoine R. Dumont (@ardumont)	package.loader: Add configuration check step within the constructor	Sep 18 2019, 12:28 PM
2efaa062ca47	db4b75e130e7	e1431d86200c	Antoine R. Dumont (@ardumont)	package.loader: Rename BaseLoader to PackageLoader	Sep 18 2019, 12:27 PM
e1431d86200c	f0a182e8fe7a	0391bfc59ea2	Antoine R. Dumont (@ardumont)	package.loader: WIP: bare base loader (Show More…)	Sep 17 2019, 3:28 PM
0391bfc59ea2	1cfd7cd92c36	beeddebb9562	Antoine R. Dumont (@ardumont)	swh.loader.package: Rename loader.py to gnu.py	Sep 16 2019, 1:45 PM

Diff 7628

View Options

MANIFEST.in

View Options

mypy.ini

View Options

requirements-swh.txt

View Options

requirements-test.txt

View Options

requirements.txt

View Options

swh/loader/package/init.py

View Options

swh/loader/package/archive.py

View Options

swh/loader/package/debian.py

View Options

swh/loader/package/deposit.py

View Options

swh/loader/package/loader.py

View Options

swh/loader/package/npm.py

View Options

swh/loader/package/pypi.py

View Options

swh/loader/package/tasks.py

View Options

swh/loader/package/tests/common.py

View Options

swh/loader/package/tests/conftest.py

View Options

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-3.diff.gz

View Options

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-3.dsc

View Options

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-4.diff.gz

View Options

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-4.dsc

View Options

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2.orig.tar.gz

View Options

swh/loader/package/tests/data/http_deb.debian.org/onefile.txt

View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/1_private_666_meta

View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/1_private_666_raw

View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/hello_2.10.json

View Options

swh/loader/package/tests/data/https_deposit.softwareheritage.org/hello_2.10.orig.tar.gz

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.tar.gz

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.2.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.3.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.4.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/nexter-1.1.0.tar.gz

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/nexter-1.1.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_70_97_c49fb8ec24a7aaab54c3dbfbb5a6ca1431419d9ee0f6c363d9ad01d2b8b1_0805nexter-1.3.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip_visit1

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip

View Options

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip_visit1

View Options

swh/loader/package/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.1.0.tar.gz

View Options

swh/loader/package/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.2.0.tar.gz

View Options

swh/loader/package/tests/data/https_pypi.org/pypi_0805nexter_json

View Options

swh/loader/package/tests/data/https_pypi.org/pypi_0805nexter_json_visit1

View Options

swh/loader/package/tests/data/https_pypi.org/pypi_nexter_json

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.2.tgz

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.3.tgz

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.4.tgz

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.5.tgz

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.1.0.tgz

View Options

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.2.0.tgz

View Options

swh/loader/package/tests/data/https_replicate.npmjs.com/org

View Options

swh/loader/package/tests/data/https_replicate.npmjs.com/org_visit1

View Options

swh/loader/package/tests/resources/tarballs/8sync-0.1.0.tar.gz

View Options

swh/loader/package/tests/test_archive.py

View Options

swh/loader/package/tests/test_common.py

View Options

swh/loader/package/tests/test_conftest.py

View Options

swh/loader/package/tests/test_debian.py

View Options

swh/loader/package/tests/test_deposit.py

View Options

swh/loader/package/tests/test_loader.py

View Options

swh/loader/package/tests/test_npm.py

View Options

swh/loader/package/tests/test_pypi.py

View Options

swh/loader/package/tests/test_tasks.py

View Options

swh/loader/package/tests/test_utils.py

View Options

swh/loader/package/utils.py

View Options

Land new package loader implementationsClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 7628

MANIFEST.in

mypy.ini

requirements-swh.txt

requirements-test.txt

requirements.txt

swh/loader/package/__init__.py

swh/loader/package/archive.py

swh/loader/package/debian.py

swh/loader/package/deposit.py

swh/loader/package/loader.py

swh/loader/package/npm.py

swh/loader/package/pypi.py

swh/loader/package/tasks.py

swh/loader/package/tests/common.py

swh/loader/package/tests/conftest.py

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-3.diff.gz

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-3.dsc

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-4.diff.gz

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-4.dsc

swh/loader/package/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2.orig.tar.gz

swh/loader/package/tests/data/http_deb.debian.org/onefile.txt

swh/loader/package/tests/data/https_deposit.softwareheritage.org/1_private_666_meta

swh/loader/package/tests/data/https_deposit.softwareheritage.org/1_private_666_raw

swh/loader/package/tests/data/https_deposit.softwareheritage.org/hello_2.10.json

swh/loader/package/tests/data/https_deposit.softwareheritage.org/hello_2.10.orig.tar.gz

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.tar.gz

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.2.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.3.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/0805nexter-1.4.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/nexter-1.1.0.tar.gz

swh/loader/package/tests/data/https_files.pythonhosted.org/nexter-1.1.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_70_97_c49fb8ec24a7aaab54c3dbfbb5a6ca1431419d9ee0f6c363d9ad01d2b8b1_0805nexter-1.3.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip_visit1

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip

swh/loader/package/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip_visit1

swh/loader/package/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.1.0.tar.gz

swh/loader/package/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.2.0.tar.gz

swh/loader/package/tests/data/https_pypi.org/pypi_0805nexter_json

swh/loader/package/tests/data/https_pypi.org/pypi_0805nexter_json_visit1

swh/loader/package/tests/data/https_pypi.org/pypi_nexter_json

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.2.tgz

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.3.tgz

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.4.tgz

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.0.5.tgz

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.1.0.tgz

swh/loader/package/tests/data/https_registry.npmjs.org/org_-_org-0.2.0.tgz

swh/loader/package/tests/data/https_replicate.npmjs.com/org

swh/loader/package/tests/data/https_replicate.npmjs.com/org_visit1

swh/loader/package/tests/resources/tarballs/8sync-0.1.0.tar.gz

swh/loader/package/tests/test_archive.py

swh/loader/package/tests/test_common.py

swh/loader/package/tests/test_conftest.py

swh/loader/package/tests/test_debian.py

swh/loader/package/tests/test_deposit.py

swh/loader/package/tests/test_loader.py

swh/loader/package/tests/test_npm.py

swh/loader/package/tests/test_pypi.py

swh/loader/package/tests/test_tasks.py

swh/loader/package/tests/test_utils.py

swh/loader/package/utils.py

tox.ini

Land new package loader implementations
ClosedPublic
Actions

Revision Contents
Changeset List

swh/loader/package/init.py