Differential D352

Bootstrap gitlab lister
ClosedPublic
Actions

Authored by ardumont on Jul 3 2018, 12:42 PM.

Tags

None

Subscribers

None

Details

Reviewers

None

Group Reviewers

Commits

rDLSd52089154775: swh.lister.core.paging_lister: Adding comments
rDLS847a8d341a8b: swh.lister.gitlab: Add Incremental lister behavior
rDLSccd0525c9b5f: swh.lister: Do not hardcode the index notion into parameter names
rDLS4c4aa0ead2c1: swh.lister: Make LISTER_NAME a class attribute
rDLSb6c5865ab17d: swh.lister.paging_lister: Improve lister's base class name
rDLSa51c36194e4a: swh.lister.gitlab: Add full gitlab lister
rDLS581028cfc5a6: swh.lister.cli: Fix cli docstring
rDLS7954e03627d3: swh.lister: Document swh.lister.tasks's intent
rDLSba146376d66e: swh.lister: Add tests around the gitlab lister
rDLSe1a460caa5d2: swh.lister.gitlab: Improve docstring
rDLS1fd44207eb44: README: Add a run section to demonstrate how to run a github lister
rDLS79cd00737f80: swh.lister.gitlab: Remove TODO about the 403 response code
rDLS3ca566776fe2: swh.lister.gitlab: Make rate limit check optional
rDLSdb36c499fe97: swh.lister.gitlab: Do not store information we cannot have
rDLS935b9cd24f45: swh.lister.core: Make gitlab lister a paging lister instance
rDLS9c5963f0155e: swh.lister.gitlab: Respect camel case convention
rDLS3760708897fc: swh.lister.gitlab: Use one configuration for the gitlab lister
rDLSd9d582f7ebdd: swh.lister.gitlab: Change the default sort order using id
rDLS8ad70b3d60ea: swh.lister.gitlab: Make the lister instance parametric
rDLSf8055a34afc9: swh.lister.gitlab: Add notes around current limitations
rDLS7a0e3a237964: gitlab.lister: Override filtering as it's not possible as is
rDLS1335a5b25f69: swh.lister.gitlab: Use gitlab's index identifier (page id)
rDLS0e517b614334: swh.lister.gitlab: Use http_url_to_repo key to build the origin url
rDLS3e62bc867ea4: swh.lister.cli: Simplify cli
rDLSafcd6997c427: swh.lister.cli: Add a basic cli to deal with create/drop db actions

Summary

Nothing changes between D351 and this except for:

separating the documentation cleanup patches
D351 got closed wrongly

There still remains to take into account the good remarks D351#6877
and D351#6882.

835c605 * lister-gitlab swh.lister.gitlab: Add notes around current limitations
15d3feb * swh.lister.gitlab: Use http_url_to_repo key to build the origin url
0528603 * gitlab.lister: Override filtering as it's not possible as is
f5b35a9 * swh.lister.gitlab: Use gitlab's index identifier (page id)
49439ab * swh.lister.cli: Add a basic cli to deal with create/drop db actions
0f9ba7b * swh.lister.gitlab: Bootstrap gitlab lister

Related D351
Related T989

Test Plan

To be added

Diff Detail

Repository

Branch

lister-gitlab

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 1269
Build 1613: arc lint + arc unit

Event Timeline

ardumont created this revision.Jul 3 2018, 12:42 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJul 3 2018, 12:42 PM

Harbormaster completed remote builds in B1268: Diff 1126.Jul 3 2018, 12:43 PM

Update:

swh.lister.gitlab: Respect camel case convention
swh.lister.gitlab: Make the lister instance parametric
swh.lister.gitlab: Make the lister's task instance parametric
swh.lister.gitlab: Change the default sort order using id
swh.lister.gitlab: Do not store information we cannot have

Harbormaster completed remote builds in B1269: Diff 1127.Jul 3 2018, 4:33 PM

Updates:

swh.lister.gitlab: Change the default sort order using id
swh.lister.gitlab: Do not store information we cannot have
swh.lister.core: Make gitlab lister a paging lister instance

TODO: Add tests

Harbormaster completed remote builds in B1270: Diff 1128.Jul 3 2018, 6:09 PM

Updates:

swh.lister.core: Make gitlab lister a paging lister instance
README: Add a run section to demonstrate how to run a github lister
swh.lister.gitlab: Remove TODO about the 403 response code
swh.lister.gitlab: Use one configuration for the gitlab lister
swh.lister.gitlab: Make rate limit check optional
README.md: Add a section to demo how to run the gitlab lister
swh.lister.gitlab: Improve docstring
swh.lister: Add tests around the gitlab lister
swh.lister: Document swh.lister.tasks's intent

Manual tests:

Check gitlab instances work [7/7]
loading of some of the origins listed by those gitlab instances (through local scheduler/listener/runner/celery)

Remains to do:

Add incremental/refresh policy for the gitlab lister
Properly separate the indexing/paging tests (cf. FIXME)

Harbormaster completed remote builds in B1271: Diff 1129.Jul 4 2018, 4:38 PM

swh.lister.gitlab: Add full gitlab lister

Harbormaster completed remote builds in B1272: Diff 1130.Jul 4 2018, 6:58 PM

Clean up gitlab lister part not related to the bootstrap

Harbormaster completed remote builds in B1277: Diff 1136.Jul 6 2018, 3:55 PM

ardumont mentioned this in D363: swh.lister.gitlab: Make the lister instance parametric.Jul 11 2018, 11:01 AM

Rebase (and adapt) according to review from D356 to D376:

swh.lister.cli: Simplify cli
swh.lister.gitlab: Use gitlab's index identifier (page id)
gitlab.lister: Override filtering as it's not possible as is
swh.lister.gitlab: Use http_url_to_repo key to build the origin url
swh.lister.gitlab: Add notes around current limitations
swh.lister.gitlab: Respect camel case convention
swh.lister.gitlab: Make the lister instance parametric
swh.lister.gitlab: Use one configuration for the gitlab lister
swh.lister.gitlab: Make the lister's task instance parametric
swh.lister.gitlab: Change the default sort order using id
swh.lister.gitlab: Do not store information we cannot have
swh.lister.core: Make gitlab lister a paging lister instance
README: Add a run section to demonstrate how to run a github lister
swh.lister.gitlab: Remove TODO about the 403 response code
swh.lister.gitlab: Make rate limit check optional
README.md: Add a section to demo how to run the gitlab lister
swh.lister.gitlab: Improve docstring
swh.lister: Add tests around the gitlab lister
swh.lister: Document swh.lister.tasks's intent
swh.lister.gitlab: Add full gitlab lister
swh.lister.cli: Fix cli docstring
swh.lister: Make LISTER_NAME a class attribute
swh.lister.paging_lister: Improve lister's base class name
swh.lister: Do not hardcode the index notion into parameter names
swh.lister.gitlab: Add Incremental lister behavior
swh.lister.gitlab: Improve headers extraction
swh.lister.gitlab.tasks: Remove spurious comma
swh.lister.core.paging_lister: Adding comments
swh.lister.gitlab: Add the presence check for the incremental lister
swh.lister.gitlab.tests: Separate properly tests per lister
swh.lister.gitlab: Read next page from headers

Harbormaster completed remote builds in B1319: Diff 1179.Jul 12 2018, 12:31 PM

Fix multiple issues

swh.lister.gitlab: Remove indexable column from gitlab lister
swh.lister.gitlab: Change uid format
swh.lister.gitlab: Fix the total pages reading instruction
swh.lister.gitlab.tasks: Fix range computations
swh.lister.core.paging_lister: Fix page initialization

Tests are fine for:

range lister
full lister
incremental lister

Harbormaster completed remote builds in B1320: Diff 1180.Jul 12 2018, 3:20 PM

swh.lister.gitlab: Add Incremental lister behavior

Made a mistake in the arc runes...

(forgot that once a diff is opened for a branch, you must specify that
you want to create a new one...)

swh.lister.gitlab: Improve headers extraction
swh.lister.gitlab.tasks: Remove spurious comma
swh.lister.core.paging_lister: Adding comments
swh.lister.gitlab: Add the presence check for the incremental lister
swh.lister.gitlab.tests: Separate properly tests per lister
swh.lister.gitlab: Read next page from headers
swh.lister.gitlab: Remove indexable column from gitlab lister
swh.lister.gitlab: Change uid format
swh.lister.gitlab: Fix the total pages reading instruction
swh.lister.gitlab.tasks: Fix range computations
swh.lister.core.paging_lister: Fix page initialization

Harbormaster completed remote builds in B1326: Diff 1186.Jul 17 2018, 2:15 PM

Harbormaster completed remote builds in B1327: Diff 1187.

This revision was not accepted when it landed; it landed in state Needs Review.Jul 17 2018, 2:16 PM

Closed by commit rDLSafcd6997c427: swh.lister.cli: Add a basic cli to deal with create/drop db actions (authored by ardumont). · Explain Why

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Path

Size

2 lines

swh/

lister/

bitbucket/

4 lines

98 lines

core/

5 lines

21 lines

debian/

6 lines

github/

4 lines

gitlab/

111 lines

17 lines

29 lines

Diff 1127

README.md

Loading...

swh/lister/bitbucket/tasks.py

Loading...

swh/lister/cli.py

Loading...

swh/lister/core/models.py

Loading...

swh/lister/core/tasks.py

Loading...

swh/lister/debian/tasks.py

Loading...

swh/lister/github/tasks.py

Loading...

swh/lister/gitlab/init.py

Loading...

swh/lister/gitlab/lister.py

Loading...

swh/lister/gitlab/models.py

Loading...

swh/lister/gitlab/tasks.py

Loading...