Differential D2028

[wip] package-loader: Implement a common package loader mechanism
ClosedPublic
Actions

Authored by ardumont on Sep 24 2019, 10:05 AM.

Details

Reviewers

None

Group Reviewers

Reviewers

Commits

rDLDBASE76c93a66371b: package.loaders: Fix type checking warnings
rDLDBASEb316d36f5d48: loader: Deal with loading error

Summary

This also demonstrate the following reimplementation of:

gnu
pypi
npm (not yet)

Note:

This will get split into multiple diffs.
This will also probably get moved out of swh-loader-core.
In the mean time, to demonstrate progress has been done, here is the wip diff.
Also, this allows to check everything is fine regarding the ci

Related T1389

Test Plan

tox

Diff Detail

Repository

rDLDBASE Generic VCS/Package Loader

Branch

package-loader

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 7922
Build 11408: tox-on-jenkins	Jenkins
Build 11407: arc lint + arc unit

Event Timeline

ardumont created this revision.Sep 24 2019, 10:05 AM

ardumont planned changes to this revision.Sep 24 2019, 10:06 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/165/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/165/console

Harbormaster failed remote builds in B7922: Diff 6808!Sep 24 2019, 10:06 AM

ardumont retitled this revision from package-loader: Implement a common package loader mechanism to [wip] package-loader: Implement a common package loader mechanism.Sep 24 2019, 10:06 AM

ardumont edited the summary of this revision. (Show Details)

ardumont edited the summary of this revision. (Show Details)Sep 24 2019, 10:08 AM

ardumont added projects: Core Loader, Origin-GNU, PyPI loader, Npm loader.

This revision was not accepted when it landed; it landed in state Changes Planned.Sep 24 2019, 2:37 PM

This revision was landed with ongoing or failed builds.

Closed by commit rDLDBASEb316d36f5d48: loader: Deal with loading error (authored by ardumont). · Explain Why

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDLDBASEb316d36f5d48: loader: Deal with loading error.

ardumont added a commit: rDLDBASE76c93a66371b: package.loaders: Fix type checking warnings.

roh, i did not want to close it...
I just wanted to update the diff... oh, well!

Revision Contents
Changeset List

Path

Size

requirements.txt

2 lines

resources/

loader.yml

3 lines

swh/

loader/

package/

19 lines

175 lines

618 lines

191 lines

5 lines

tests/

common.py

86 lines

resources/

files.pythonhosted.org/

packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip

1 line

packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip

1 line

	ftp.gnu.org/
	tarballs/

	gnu_8sync_8sync-0.1.0.tar.gz
	8sync-0.1.0.tar.gz

json/

0805nexter+new-made-up-release.json

114 lines

0805nexter-unpublished-release.json

116 lines

pypi.org/

pypi_0805nexter_json

95 lines

tarballs/

210 lines

313 lines

72 lines

59 lines

2 lines

Commit	Tree	Parents	Author	Summary	Date
76c93a66371b	7cbe5e78cd12	b316d36f5d48	Antoine R. Dumont (@ardumont)	package.loaders: Fix type checking warnings	Sep 23 2019, 6:41 PM
b316d36f5d48	20175f8b52c1	664b29a2a9da	Antoine R. Dumont (@ardumont)	loader: Deal with loading error	Sep 23 2019, 6:24 PM
664b29a2a9da	5e316f4e2e07	926c3171af64	Antoine R. Dumont (@ardumont)	gnu: Improve yet again the version matching regexp	Sep 23 2019, 4:26 PM
926c3171af64	8245968b8b6b	a4177221972d	Antoine R. Dumont (@ardumont)	gnu: Improve slightly release number extraction	Sep 23 2019, 3:45 PM
a4177221972d	b62c5e9d1ce5	735c5e62afe3	Antoine R. Dumont (@ardumont)	gnu: Refactor using PackageLoader as basis	Sep 23 2019, 1:32 PM
735c5e62afe3	836e71bf9f53	c42079e5063e	Antoine R. Dumont (@ardumont)	package.pypi: Extract download function into its own utils module	Sep 23 2019, 10:22 AM
c42079e5063e	87a6e688ba40	43e88116be4a	Antoine R. Dumont (@ardumont)	package.loader: Fix typo	Sep 23 2019, 10:19 AM
43e88116be4a	c16c5eb93441	3e6bd5630580	Antoine R. Dumont (@ardumont)	package.loader.load: Add missing type annotation	Sep 23 2019, 10:05 AM
3e6bd5630580	c4fb897bd41a	10b09d0b6b05	Antoine R. Dumont (@ardumont)	package.loader: Transform FIXME into a comment	Sep 23 2019, 10:05 AM
10b09d0b6b05	d6c48b38b7ce	89a087201074	Antoine R. Dumont (@ardumont)	package.loader: Add notes about what's not yet implemented	Sep 23 2019, 10:05 AM
89a087201074	df5b419089e7	acde74c2bd08	Antoine R. Dumont (@ardumont)	package.pypi: Make PyPIClient a function (Show More…)	Sep 22 2019, 9:31 AM
acde74c2bd08	52d7b4bd2f99	ba245e681952	Antoine R. Dumont (@ardumont)	test_pypi: Fix pep8 violation	Sep 22 2019, 9:30 AM
ba245e681952	afee9a7c26f9	4a437bb48b7a	Antoine R. Dumont (@ardumont)	package.pypi: Add hash checks	Sep 20 2019, 4:32 PM
4a437bb48b7a	769d8deaa8ce	098005605a37	Antoine R. Dumont (@ardumont)	package.pypi: Clean up dead code	Sep 20 2019, 4:03 PM
098005605a37	93fb223050e9	4086cab9fe6a	Antoine R. Dumont (@ardumont)	package.pypi: Add one loading use case and make the loader work (Show More…)	Sep 20 2019, 4:01 PM
4086cab9fe6a	07252c02e874	2df3bd9e1c85	Antoine R. Dumont (@ardumont)	package.pypi: Add tests around download function	Sep 20 2019, 11:55 AM
2df3bd9e1c85	aa6f8b366394	ac5f1a878a16	Antoine R. Dumont (@ardumont)	loader.pypi: Add tests around sdist_parse	Sep 20 2019, 7:40 AM
ac5f1a878a16	6368ceea413e	b2752142cc04	Antoine R. Dumont (@ardumont)	loader.pypi: Add test around sdist_parse	Sep 20 2019, 7:32 AM
b2752142cc04	c7dd0b4f52b6	902845193523	Antoine R. Dumont (@ardumont)	pypi: Fix pep8 violations	Sep 20 2019, 7:30 AM
902845193523	8c3ce88545ed	26190a555ad8	Antoine R. Dumont (@ardumont)	package.pypi: Add tests around PyPIClient	Sep 20 2019, 12:05 AM
26190a555ad8	3b8e4f75fc3a	f4c44e49bb3e	Antoine R. Dumont (@ardumont)	package.pypi: Update docstrings and type annotations	Sep 19 2019, 10:40 PM
f4c44e49bb3e	3b206586fe0d	03f5a1f90ea9	Antoine R. Dumont (@ardumont)	package.pypi: Add type annotation	Sep 19 2019, 9:44 PM
03f5a1f90ea9	ed048971350f	25aac0ea9f59	Antoine R. Dumont (@ardumont)	test_pypi: Add tests around `author` function	Sep 19 2019, 9:39 PM
25aac0ea9f59	e80404fe6309	64ba90a75197	Antoine R. Dumont (@ardumont)	loader.package: Bootstrap tests	Sep 19 2019, 9:33 PM
64ba90a75197	0bbb7445fdd3	102bc7b38c34	Antoine R. Dumont (@ardumont)	loader.package: Fix pep8 violations	Sep 19 2019, 9:33 PM
102bc7b38c34	1ff26dbefc55	9a456a652ac7	Antoine R. Dumont (@ardumont)	package.pypi: Fix __init__ call	Sep 19 2019, 9:32 PM
9a456a652ac7	7b1f2cf7e952	a62355d56656	Antoine R. Dumont (@ardumont)	package.loader: Fix import	Sep 19 2019, 9:19 PM
a62355d56656	32342ae372e3	3f3e774c47bb	Antoine R. Dumont (@ardumont)	package.loader: Remove spurious fixme	Sep 19 2019, 9:19 PM
3f3e774c47bb	bcdb48b19a0c	fc977e6e92cb	Antoine R. Dumont (@ardumont)	requirements: Add missing pkginfo	Sep 19 2019, 9:18 PM
fc977e6e92cb	fd9f9c9f04c8	a2f2d0f88d40	Antoine R. Dumont (@ardumont)	wip: Add pypi implementation	Sep 19 2019, 4:39 PM
a2f2d0f88d40	2f4607e89e91	8d46a9613603	Antoine R. Dumont (@ardumont)	package.loader: Implement further pypi loader	Sep 18 2019, 6:21 PM
8d46a9613603	3eb2b1db9564	e42a7c6519c1	Antoine R. Dumont (@ardumont)	package.loader: clarify method name intent	Sep 18 2019, 6:21 PM
e42a7c6519c1	366bbfb820e8	99fd2584ea4a	Antoine R. Dumont (@ardumont)	package.loader: Fix method name	Sep 18 2019, 6:21 PM
99fd2584ea4a	59d946225e4c	37b4d313f580	Antoine R. Dumont (@ardumont)	loader.package: Add basic pypi collaborators	Sep 18 2019, 4:28 PM
37b4d313f580	d325c3b04e44	2ad84f608037	Antoine R. Dumont (@ardumont)	package.loader: Simplify message computation	Sep 18 2019, 3:18 PM
2ad84f608037	0a10135a38cd	51d742eb278c	Antoine R. Dumont (@ardumont)	package.loader: Check only configuration	Sep 18 2019, 3:18 PM
51d742eb278c	13f7839baba6	2429a0aac752	Antoine R. Dumont (@ardumont)	package.loader: Add FIXME around exceptions handling	Sep 18 2019, 2:50 PM
2429a0aac752	9f851d6de882	f4d44cd9c45c	Antoine R. Dumont (@ardumont)	package.loader: Improve variable name (Show More…)	Sep 18 2019, 2:43 PM
f4d44cd9c45c	9ac0c9b1d710	271473fb553d	Antoine R. Dumont (@ardumont)	package.loader: No parent revision by default	Sep 18 2019, 2:41 PM
271473fb553d	0f9a48b5e0a1	e1a74f7171d6	Antoine R. Dumont (@ardumont)	loader.package: Deal with origin and visit policy	Sep 18 2019, 2:34 PM
e1a74f7171d6	06d200ff2f94	e3387de22d16	Antoine R. Dumont (@ardumont)	loader.package: Add docstrings to methods	Sep 18 2019, 12:28 PM
e3387de22d16	08a3200452d7	839bc2d44e7d	Antoine R. Dumont (@ardumont)	package.loader: Add FIXME about origin/visit creation/update	Sep 18 2019, 12:28 PM
839bc2d44e7d	83f1437ed1ed	2efaa062ca47	Antoine R. Dumont (@ardumont)	package.loader: Add configuration check step within the constructor	Sep 18 2019, 12:28 PM
2efaa062ca47	db4b75e130e7	e1431d86200c	Antoine R. Dumont (@ardumont)	package.loader: Rename BaseLoader to PackageLoader	Sep 18 2019, 12:27 PM
e1431d86200c	f0a182e8fe7a	0391bfc59ea2	Antoine R. Dumont (@ardumont)	package.loader: WIP: bare base loader (Show More…)	Sep 17 2019, 3:28 PM
0391bfc59ea2	1cfd7cd92c36	beeddebb9562	Antoine R. Dumont (@ardumont)	swh.loader.package: Rename loader.py to gnu.py	Sep 16 2019, 1:45 PM