Differential D411

pypi.loader: Finalize pypi loader (take 2)
AbandonedPublic
Actions

Authored by ardumont on Sep 18 2018, 11:11 AM.

Details

Reviewers

olasd

Summary

Related T421
Related D408 (mistakenly closed)

Test Plan

make test

Diff Detail

Repository

rDLDPY PyPI loader

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 1394
Build 1738: arc lint + arc unit

Event Timeline

ardumont created this revision.Sep 18 2018, 11:11 AM

Harbormaster completed remote builds in B1394: Diff 1284.Sep 18 2018, 11:11 AM

\o/

This revision is now accepted and ready to land.Sep 18 2018, 11:14 AM

ardumont added inline comments.Sep 18 2018, 11:19 AM

swh/loader/pypi/loader.py
148	11:10 <+olasd> currently the main issue is that incremental loading isn't incremental 11:10 <+olasd> because the list of old versions is stored as an iterator, which gets consumed by the first check and then never returns again :p Is this what you are referring to?

olasd added a child revision: D412: Add trivial command line for the PyPI loader.Sep 18 2018, 11:19 AM

@ardumont this probably should be abandoned as it didn't auto-close...

@ardumont this probably should be abandoned as it didn't auto-close...

Mmm, right.

Revision Contents
Changeset List

Path

Size

README.md

100 lines

debian/

2 lines

9 lines

1 line

1 line

3 lines

4 lines

swh/

loader/

__init__.py

1 line

pypi/

1 line

442 lines

70 lines

244 lines

10 lines

19 lines

tests/

common.py

151 lines

resources/

1 line

78 lines

113 lines

259 lines

	pypi/	tests/

__init__.py

Commit	Tree	Parents	Author	Summary	Date
087c50f73b98	412f9c4fa5e7	17904b456f4b	Antoine R. Dumont (@ardumont)	README: Describes the PyPI loader (Show More…)	Sep 17 2018, 3:11 PM
17904b456f4b	19afdb6d9e62	3dce19b02f03	Antoine R. Dumont (@ardumont)	pypi.client: Simplify hashes computation	Sep 17 2018, 12:26 PM
3dce19b02f03	3f3ead37ddad	7b84f4bc06d9	Antoine R. Dumont (@ardumont)	pypi.client: Fix length parameter passing	Sep 15 2018, 1:04 AM
7b84f4bc06d9	c74ddd35e9ec	1190b3644c7d	Antoine R. Dumont (@ardumont)	pypi.client: Use new MultiHash.from_path method (Show More…)	Sep 15 2018, 12:59 AM
1190b3644c7d	be70166ae779	bcc34fdcfe6e	Antoine R. Dumont (@ardumont)	pypi.client: Improve variable name which clarifies intent	Sep 14 2018, 11:43 PM
bcc34fdcfe6e	088cb6fa6354	84f0de41b63f	Antoine R. Dumont (@ardumont)	pypi.client: Simplify the hashes computation code	Sep 14 2018, 11:42 PM
84f0de41b63f	e97e0b0ee2bb	27ea42192186	Antoine R. Dumont (@ardumont)	pypi.client: Update according to latest swh.model.hashutil api (Show More…)	Sep 14 2018, 8:09 PM
27ea42192186	504e52683ff5	7b6560bd469d	Antoine R. Dumont (@ardumont)	pypi.client: Compute hashes and write the tarball in one roundtrip (Show More…)	Sep 14 2018, 1:27 AM
7b6560bd469d	1147df331d8a	37bf4601b851	Antoine R. Dumont (@ardumont)	pypi.tasks: Rename to LoadPyPI task	Sep 14 2018, 12:06 AM
37bf4601b851	30c07fc2844b	a8f8272ffa27	Antoine R. Dumont (@ardumont)	pypi.client: Improve PyPIProject docstring	Sep 13 2018, 11:59 PM
a8f8272ffa27	681cd5672954	0171a108b3c3	Antoine R. Dumont (@ardumont)	pypi.loader: Make loader fully incremental in loading swh objects (Show More…)	Sep 13 2018, 11:38 PM
0171a108b3c3	5de23930fb69	45d6035465db	Antoine R. Dumont (@ardumont)	loader.pypi: Clarify term from release to artifact	Sep 13 2018, 8:16 PM
45d6035465db	e88e701c2f40	830255622a2a	Antoine R. Dumont (@ardumont)	pypi.client: Clarify filtering release artifacts method	Sep 13 2018, 8:09 PM
830255622a2a	4c466543e271	dab7264bb0cf	Antoine R. Dumont (@ardumont)	test_loader: Fix pep8 violation	Sep 13 2018, 8:08 PM
dab7264bb0cf	d8dcb7a0aa4e	aa66e640723e	Antoine R. Dumont (@ardumont)	pypi.client: Dissociate artifact and release object	Sep 13 2018, 8:00 PM
aa66e640723e	4c53b0000103	a2af496de57d	Antoine R. Dumont (@ardumont)	pypi.client: Fix wrong verb use (allow instead of permit)	Sep 13 2018, 7:25 PM
a2af496de57d	e6b718fd17be	0837698a5b86	Antoine R. Dumont (@ardumont)	*: Use less _ in prefix variable names	Sep 13 2018, 7:25 PM
0837698a5b86	f1e325c3ac2d	6ed47d3932c2	Antoine R. Dumont (@ardumont)	pypi.loader: Remove (override) comment	Sep 13 2018, 7:16 PM
6ed47d3932c2	0f98e0c1fc10	9cd6aa90baaf	Antoine R. Dumont (@ardumont)	*: Use "PyPI" terminology consistently everywhere	Sep 13 2018, 7:10 PM
9cd6aa90baaf	80ce98f31c68	4459927dd167	Antoine R. Dumont (@ardumont)	Move PyPiProject from model to client module	Sep 13 2018, 7:09 PM
4459927dd167	40bade6bc6eb	c86b4124fad9	Antoine R. Dumont (@ardumont)	test_loader: Fix typo	Sep 13 2018, 7:04 PM
c86b4124fad9	392a8c39208e	beecf08d77cf	Antoine R. Dumont (@ardumont)	loader.pypi: Move converter functions in converters module	Sep 13 2018, 7:03 PM
beecf08d77cf	2007cad98544	f3fcab87ff66	Antoine R. Dumont (@ardumont)	pypi.client: Clarify the term from release to artifact (Show More…)	Sep 13 2018, 6:40 PM
f3fcab87ff66	9b504a9542de	90a29dca07de	Antoine R. Dumont (@ardumont)	pypi.client: Rename prepare_release_artifact method	Sep 13 2018, 6:24 PM
90a29dca07de	4b57914bc328	fc2de869046e	Antoine R. Dumont (@ardumont)	pypi.client: Extract PyPI instance's base url as instance attribute	Sep 13 2018, 6:18 PM
fc2de869046e	650cadda4b45	ada276379d9d	Antoine R. Dumont (@ardumont)	pypi.client: Improve error message when failing to fetch artifacts (Show More…)	Sep 13 2018, 6:17 PM
ada276379d9d	f66f630d2b44	f3967ea4b09c	Antoine R. Dumont (@ardumont)	loader.pypi.client: Improve docstrings (Show More…)	Sep 13 2018, 6:16 PM
f3967ea4b09c	af04dc4de996	396493e233af	Antoine R. Dumont (@ardumont)	tests.common: Improve docstring	Sep 11 2018, 11:24 AM
396493e233af	472c3c327858	0c63f61a1a86	Antoine R. Dumont (@ardumont)	loader.pypi: Add loading origin without changes scenario (Show More…)	Sep 7 2018, 7:28 PM
0c63f61a1a86	31f4ffea31da	0c2f2270eba1	Antoine R. Dumont (@ardumont)	loader.pypi: Add integration test on main loader use case (Show More…)	Sep 7 2018, 7:10 PM
0c2f2270eba1	8382950ec6fd	ae2b73b87d84	Antoine R. Dumont (@ardumont)	pypi.model.tests: Bad author field stays as is (Show More…)	Sep 7 2018, 2:03 PM
ae2b73b87d84	45403c2e8085	920a49d74b9c	Antoine R. Dumont (@ardumont)	Ignore swh/loader/pypi/_version.py generated file	Sep 7 2018, 11:01 AM
920a49d74b9c	fad9e6ac0bd7	b80666fd24b8	Antoine R. Dumont (@ardumont)	d/changelog: Fix typo in package name	Sep 7 2018, 11:01 AM
b80666fd24b8	a1ab2b3467de	620553b72cfa	Antoine R. Dumont (@ardumont)	pypi.model: When no pkg-info, skip the release artifact (Show More…)	Sep 6 2018, 2:24 PM
620553b72cfa	bfcde755df65	fad64ebd7f1b	Antoine R. Dumont (@ardumont)	pypi.tests: Fix missing parameter	Sep 6 2018, 2:24 PM
fad64ebd7f1b	e87dd767e7a7	a2395b6e22d7	Antoine R. Dumont (@ardumont)	d/*: Add pkginfo dependency	Sep 6 2018, 1:42 PM
a2395b6e22d7	eaca335dfad1	ad7a684cffbb	Antoine R. Dumont (@ardumont)	pypi.loader: Make the loader incremental (Show More…)	Sep 5 2018, 3:58 PM
ad7a684cffbb	88b581cca7a5	a92beec47527	Antoine R. Dumont (@ardumont)	pypi.model: Clarify warning log	Sep 5 2018, 11:49 AM
a92beec47527	5c49b7404c5f	a4b9ab48529e	Antoine R. Dumont (@ardumont)	pypi.model: Fix author information when metadata coming from pypi	Sep 4 2018, 2:20 PM
a4b9ab48529e	4fc4dcee59e1	9d0dca3b6b9a	Antoine R. Dumont (@ardumont)	pypi.client: Improve error message during hash mismatch	Sep 1 2018, 7:36 PM
9d0dca3b6b9a	b20d817ccffd	6832e00c2a66	Antoine R. Dumont (@ardumont)	loader.pypi: Do not break when author information is missing	Sep 1 2018, 7:31 PM
6832e00c2a66	7583204a3476	1fbbffd4f8d2	Antoine R. Dumont (@ardumont)	loader.pypi: Make the tests great again	Aug 30 2018, 6:54 PM
1fbbffd4f8d2	990132b3de3d	d5570e5f1aba	Antoine R. Dumont (@ardumont)	pypi.model: Use internal variable name	Aug 30 2018, 2:09 PM
d5570e5f1aba	10311134c317	15d4e24720d6	Antoine R. Dumont (@ardumont)	pypi.client: Keep the project information in the cache filename	Aug 30 2018, 1:59 PM
15d4e24720d6	4b6709e31bbb	c8bd7b44fcb0	Antoine R. Dumont (@ardumont)	pypi.client: Remove duplicated aliased entry	Aug 29 2018, 10:34 PM
c8bd7b44fcb0	0d3c6348570b	77ef29d6ddc3	Antoine R. Dumont (@ardumont)	pypi.model: Remove spurious logging instruction	Aug 29 2018, 10:24 PM
77ef29d6ddc3	ece18449f0b8	6832a031716f	Antoine R. Dumont (@ardumont)	pypi.client: Remove unneeded entry	Aug 29 2018, 10:23 PM
6832a031716f	4bfe075a010b	1df34ed78a46	Antoine R. Dumont (@ardumont)	pypi.client: Update docstring	Aug 29 2018, 10:01 PM
1df34ed78a46	1c034ca2aa2f	4d7954639ba2	Antoine R. Dumont (@ardumont)	pypi.loader: Retrieve pypi metadata information only when needed (Show More…)	Aug 29 2018, 10:00 PM
4d7954639ba2	8b14e058f754	5b6dbbd871f4	Antoine R. Dumont (@ardumont)	pypi.loader: Fix pep8 violation	Aug 29 2018, 8:27 PM
5b6dbbd871f4	9ec336e9486e	ba842797e80b	Antoine R. Dumont (@ardumont)	pypi.loader: Use pkginfo if found, fallback to pypi api if missing (Show More…)	Aug 29 2018, 8:26 PM
ba842797e80b	c45b130b9cd8	070adf03faa5	Antoine R. Dumont (@ardumont)	pypi.client: Use shutil.copyfile	Aug 29 2018, 6:11 PM
070adf03faa5	378610b01c47	571cafc89ed3	Antoine R. Dumont (@ardumont)	loader: Parse the metadata from the retrieved package (Show More…)	Aug 29 2018, 4:07 PM
571cafc89ed3	1e0d830d6ecf	a0f7271b57b8	Antoine R. Dumont (@ardumont)	client: Actually use the cache when the file is present	Aug 29 2018, 4:06 PM
a0f7271b57b8	7c658f5d451a	de38d7c7ed9e	Antoine R. Dumont (@ardumont)	pypi.loader: Use the filename as snapshot branch name (Show More…)	Aug 23 2018, 3:27 PM
de38d7c7ed9e	5d2e9b0be998	1e050abfa8ba	Antoine R. Dumont (@ardumont)	pypi.loader: Use all available sdist files provided for a release (Show More…)	Aug 23 2018, 2:31 PM
1e050abfa8ba	1ec2e55e2330	153137be1f81	Antoine R. Dumont (@ardumont)	pypi.client: Make the cache mode save the release archive (Show More…)	Aug 23 2018, 10:41 AM
153137be1f81	6f155f3c6927	3e40512f0bf7	Antoine R. Dumont (@ardumont)	pypi.loader: Fix pep8 warnings	Aug 23 2018, 10:31 AM
3e40512f0bf7	6c3b3367e0a0	d4e10d44e8db	Antoine R. Dumont (@ardumont)	pypy.loader: Use directly the revision as snapshot branch target (Show More…)	Aug 23 2018, 10:30 AM
d4e10d44e8db	00ec0332ad4e	14502fa37400	Antoine R. Dumont (@ardumont)	swh.loader.pypi: Improve readability through tuple use (Show More…)	Aug 3 2018, 9:19 PM
14502fa37400	b2acc6cb96fe	2f5463a4b783	Antoine R. Dumont (@ardumont)	swh.loader.pypi.client: Clean up intermediary files sooner	Aug 3 2018, 8:51 PM
2f5463a4b783	239198de27d7	db75c87d9cff	Antoine R. Dumont (@ardumont)	swh.loader.pypi.model: When no release is provided skip	Aug 3 2018, 1:58 PM
db75c87d9cff	869309f6c28c	74c6cf25258e	Antoine R. Dumont (@ardumont)	swh.loader.pypi.loader: Remove wrong name entry in revision	Aug 3 2018, 1:54 PM
74c6cf25258e	a39ef3400856	1f0f6c7a47d2	Antoine R. Dumont (@ardumont)	swh.loader.pypi.model: Fallback when comment text is not provided	Aug 3 2018, 1:54 PM
1f0f6c7a47d2	d805fa7b1a64	a89af4ae01de	Antoine R. Dumont (@ardumont)	swh.loader.pypi.model: Return empty author when none provided	Aug 3 2018, 1:46 PM
a89af4ae01de	580ccd009928	9f28590b261c	Antoine R. Dumont (@ardumont)	swh.loader.pypi.model: Improve release artifact election (Show More…)	Aug 3 2018, 1:29 PM
9f28590b261c	5de3dc09710c	54e7aa4f9dab	Antoine R. Dumont (@ardumont)	swh.loader.pypi: Improve temporary directory bootstrap (Show More…)	Aug 3 2018, 1:25 PM
54e7aa4f9dab	0acb698d74c3	909574f7fa56	Antoine R. Dumont (@ardumont)	loader.pypi: Improve method name to clarify intent	Aug 3 2018, 12:40 PM
909574f7fa56	f60b878d6e95	c1a3a29b423e	Antoine R. Dumont (@ardumont)	loader.pypi: Separate computations and storage instructions	Aug 3 2018, 12:34 PM
c1a3a29b423e	942795264499	d32c08b2f364	Antoine R. Dumont (@ardumont)	swh.loader.pypi: Update docstrings	Aug 3 2018, 12:34 PM
d32c08b2f364	05dc2f60978f	528636e2cb16	Antoine R. Dumont (@ardumont)	swh.loader.pypi.client: Remove dead code	Aug 3 2018, 11:56 AM
528636e2cb16	54767f53e8ef	0dd0cc0d4eb4	Antoine R. Dumont (@ardumont)	swh.loader.pypi.model: Remove dead code	Aug 3 2018, 11:44 AM
0dd0cc0d4eb4	9731edf08b41	ca6970e52f61	Antoine R. Dumont (@ardumont)	swh.loader.pypi.loader: Do not enforce any order in revision (Show More…)	Aug 2 2018, 8:55 PM
ca6970e52f61	a58fee90db49	21ff9b155a49	Antoine R. Dumont (@ardumont)	swh.loader.pypi.model: When failing to sort, use a simpler sort	Aug 2 2018, 8:44 PM
21ff9b155a49	f322b358d747	6c0db4133826	Antoine R. Dumont (@ardumont)	loader.pypi.model: Log when encountering unexpected release scheme (Show More…)	Aug 2 2018, 8:40 PM
6c0db4133826	2d706ec46120	1c039f6902eb	Antoine R. Dumont (@ardumont)	swh.loader.pypi: Fix release order	Aug 2 2018, 7:54 PM
1c039f6902eb	b00f8320c600	bc3facb3ab8e	Antoine R. Dumont (@ardumont)	swh.loader.pypi.client: Improve client's error logging	Aug 2 2018, 9:32 PM
bc3facb3ab8e	cec53fa25666	503aeadfee19	Antoine R. Dumont (@ardumont)	swh.loader.pypi.model: Skip when no release is defined	Aug 2 2018, 8:39 PM
503aeadfee19	c8ef18745be4	eccd21ae7435	Antoine R. Dumont (@ardumont)	swh.loader.pypi.client: Avoid breakage if unprovided data	Aug 2 2018, 8:29 PM
eccd21ae7435	3da0d0d27633	2d6f77b8b55e	Antoine R. Dumont (@ardumont)	swh.loader.pypi.client: Fix wrong message in exception	Aug 2 2018, 8:15 PM
2d6f77b8b55e	90a27daf27bd	46551114e1d2	Antoine R. Dumont (@ardumont)	swh.loader.pypi.model: Fix wrong indentation	Aug 2 2018, 8:31 PM
46551114e1d2	a52f9bb61bfb	44c1000a5e4d	Antoine R. Dumont (@ardumont)	swh.loader.pypi.model: Fix the wrong call to fetch_release	Aug 2 2018, 8:47 PM
44c1000a5e4d	ac61ed441124	df846eeb0dd6	Antoine R. Dumont (@ardumont)	swh.loader.pypi: Use the default loader which filter existing data (Show More…)	Aug 2 2018, 7:28 PM
df846eeb0dd6	bc90f63902c8	2f63fa52f5fc	Antoine R. Dumont (@ardumont)	swh.loader.pypi: Retrieve information per pypi release (Show More…)	Aug 2 2018, 6:51 PM
2f63fa52f5fc	9524ff27526f	d2390dc4d601	Antoine R. Dumont (@ardumont)	debian/rules: Add missing file removal instruction	Aug 2 2018, 3:59 PM
d2390dc4d601	808eb3ccca4e	6a71db661bc9	Antoine R. Dumont (@ardumont)	loader.pypi: Store pypi origin's releases in storage (Show More…)	Aug 2 2018, 2:11 PM
6a71db661bc9	fd4ae83aa840	94d353b85aeb	Antoine R. Dumont (@ardumont)	swh.loader.pypi.loader: Bootstrap pypi loader (Show More…)	Aug 1 2018, 6:23 PM