Page MenuHomeSoftware Heritage
Differential D352

Bootstrap gitlab lister
ClosedPublic
Actions

Authored by ardumont on Jul 3 2018, 12:42 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Commits
rDLSd52089154775: swh.lister.core.paging_lister: Adding comments
rDLS847a8d341a8b: swh.lister.gitlab: Add Incremental lister behavior
rDLSccd0525c9b5f: swh.lister: Do not hardcode the index notion into parameter names
rDLS4c4aa0ead2c1: swh.lister: Make LISTER_NAME a class attribute
rDLSb6c5865ab17d: swh.lister.paging_lister: Improve lister's base class name
rDLSa51c36194e4a: swh.lister.gitlab: Add full gitlab lister
rDLS581028cfc5a6: swh.lister.cli: Fix cli docstring
rDLS7954e03627d3: swh.lister: Document swh.lister.tasks's intent
rDLSba146376d66e: swh.lister: Add tests around the gitlab lister
rDLSe1a460caa5d2: swh.lister.gitlab: Improve docstring
rDLS1fd44207eb44: README: Add a run section to demonstrate how to run a github lister
rDLS79cd00737f80: swh.lister.gitlab: Remove TODO about the 403 response code
rDLS3ca566776fe2: swh.lister.gitlab: Make rate limit check optional
rDLSdb36c499fe97: swh.lister.gitlab: Do not store information we cannot have
rDLS935b9cd24f45: swh.lister.core: Make gitlab lister a paging lister instance
rDLS9c5963f0155e: swh.lister.gitlab: Respect camel case convention
rDLS3760708897fc: swh.lister.gitlab: Use one configuration for the gitlab lister
rDLSd9d582f7ebdd: swh.lister.gitlab: Change the default sort order using id
rDLS8ad70b3d60ea: swh.lister.gitlab: Make the lister instance parametric
rDLSf8055a34afc9: swh.lister.gitlab: Add notes around current limitations
rDLS7a0e3a237964: gitlab.lister: Override filtering as it's not possible as is
rDLS1335a5b25f69: swh.lister.gitlab: Use gitlab's index identifier (page id)
rDLS0e517b614334: swh.lister.gitlab: Use http_url_to_repo key to build the origin url
rDLS3e62bc867ea4: swh.lister.cli: Simplify cli
rDLSafcd6997c427: swh.lister.cli: Add a basic cli to deal with create/drop db actions
Summary

Nothing changes between D351 and this except for:

  • separating the documentation cleanup patches
  • D351 got closed wrongly

There still remains to take into account the good remarks D351#6877
and D351#6882.

835c605 * lister-gitlab swh.lister.gitlab: Add notes around current limitations
15d3feb * swh.lister.gitlab: Use http_url_to_repo key to build the origin url
0528603 * gitlab.lister: Override filtering as it's not possible as is
f5b35a9 * swh.lister.gitlab: Use gitlab's index identifier (page id)
49439ab * swh.lister.cli: Add a basic cli to deal with create/drop db actions
0f9ba7b * swh.lister.gitlab: Bootstrap gitlab lister

Related D351
Related T989

Test Plan

To be added

Diff Detail

Repository
rDLS Listers
Branch
lister-gitlab
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 1319
Build 1663: arc lint + arc unit

Event Timeline

ardumont created this revision.Jul 3 2018, 12:42 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 3 2018, 12:42 PM
Harbormaster completed remote builds in B1268: Diff 1126.Jul 3 2018, 12:43 PM
ardumont updated this revision to Diff 1127.Jul 3 2018, 4:33 PM

Update:

  • swh.lister.gitlab: Respect camel case convention
  • swh.lister.gitlab: Make the lister instance parametric
  • swh.lister.gitlab: Make the lister's task instance parametric
  • swh.lister.gitlab: Change the default sort order using id
  • swh.lister.gitlab: Do not store information we cannot have
Harbormaster completed remote builds in B1269: Diff 1127.Jul 3 2018, 4:33 PM
ardumont updated this revision to Diff 1128.Jul 3 2018, 6:09 PM

Updates:

  • swh.lister.gitlab: Change the default sort order using id
  • swh.lister.gitlab: Do not store information we cannot have
  • swh.lister.core: Make gitlab lister a paging lister instance

TODO: Add tests

Harbormaster completed remote builds in B1270: Diff 1128.Jul 3 2018, 6:09 PM
ardumont updated this revision to Diff 1129.Jul 4 2018, 4:38 PM

Updates:

  • swh.lister.core: Make gitlab lister a paging lister instance
  • README: Add a run section to demonstrate how to run a github lister
  • swh.lister.gitlab: Remove TODO about the 403 response code
  • swh.lister.gitlab: Use one configuration for the gitlab lister
  • swh.lister.gitlab: Make rate limit check optional
  • README.md: Add a section to demo how to run the gitlab lister
  • swh.lister.gitlab: Improve docstring
  • swh.lister: Add tests around the gitlab lister
  • swh.lister: Document swh.lister.tasks's intent

Manual tests:

Remains to do:

  • Add incremental/refresh policy for the gitlab lister
  • Properly separate the indexing/paging tests (cf. FIXME)
Harbormaster completed remote builds in B1271: Diff 1129.Jul 4 2018, 4:38 PM
ardumont updated this revision to Diff 1130.Jul 4 2018, 6:58 PM
  • swh.lister.gitlab: Add full gitlab lister
Harbormaster completed remote builds in B1272: Diff 1130.Jul 4 2018, 6:58 PM
ardumont updated this revision to Diff 1136.Jul 6 2018, 3:55 PM

Clean up gitlab lister part not related to the bootstrap

Harbormaster completed remote builds in B1277: Diff 1136.Jul 6 2018, 3:55 PM
ardumont mentioned this in D363: swh.lister.gitlab: Make the lister instance parametric.Jul 11 2018, 11:01 AM
ardumont updated this revision to Diff 1179.EditedJul 12 2018, 12:31 PM

Rebase (and adapt) according to review from D356 to D376:

  • swh.lister.cli: Simplify cli
  • swh.lister.gitlab: Use gitlab's index identifier (page id)
  • gitlab.lister: Override filtering as it's not possible as is
  • swh.lister.gitlab: Use http_url_to_repo key to build the origin url
  • swh.lister.gitlab: Add notes around current limitations
  • swh.lister.gitlab: Respect camel case convention
  • swh.lister.gitlab: Make the lister instance parametric
  • swh.lister.gitlab: Use one configuration for the gitlab lister
  • swh.lister.gitlab: Make the lister's task instance parametric
  • swh.lister.gitlab: Change the default sort order using id
  • swh.lister.gitlab: Do not store information we cannot have
  • swh.lister.core: Make gitlab lister a paging lister instance
  • README: Add a run section to demonstrate how to run a github lister
  • swh.lister.gitlab: Remove TODO about the 403 response code
  • swh.lister.gitlab: Make rate limit check optional
  • README.md: Add a section to demo how to run the gitlab lister
  • swh.lister.gitlab: Improve docstring
  • swh.lister: Add tests around the gitlab lister
  • swh.lister: Document swh.lister.tasks's intent
  • swh.lister.gitlab: Add full gitlab lister
  • swh.lister.cli: Fix cli docstring
  • swh.lister: Make LISTER_NAME a class attribute
  • swh.lister.paging_lister: Improve lister's base class name
  • swh.lister: Do not hardcode the index notion into parameter names
  • swh.lister.gitlab: Add Incremental lister behavior
  • swh.lister.gitlab: Improve headers extraction
  • swh.lister.gitlab.tasks: Remove spurious comma
  • swh.lister.core.paging_lister: Adding comments
  • swh.lister.gitlab: Add the presence check for the incremental lister
  • swh.lister.gitlab.tests: Separate properly tests per lister
  • swh.lister.gitlab: Read next page from headers
Harbormaster completed remote builds in B1319: Diff 1179.Jul 12 2018, 12:31 PM
ardumont updated this revision to Diff 1180.Jul 12 2018, 3:20 PM

Fix multiple issues

  • swh.lister.gitlab: Remove indexable column from gitlab lister
  • swh.lister.gitlab: Change uid format
  • swh.lister.gitlab: Fix the total pages reading instruction
  • swh.lister.gitlab.tasks: Fix range computations
  • swh.lister.core.paging_lister: Fix page initialization

Tests are fine for:

  • range lister
  • full lister
  • incremental lister
Harbormaster completed remote builds in B1320: Diff 1180.Jul 12 2018, 3:20 PM
ardumont updated this revision to Diff 1186.Jul 16 2018, 9:50 AM

swh.lister.gitlab: Add Incremental lister behavior

ardumont updated this revision to Diff 1187.Jul 16 2018, 9:53 AM

Made a mistake in the arc runes...

(forgot that once a diff is opened for a branch, you must specify that
you want to create a new one...)

  • swh.lister.gitlab: Improve headers extraction
  • swh.lister.gitlab.tasks: Remove spurious comma
  • swh.lister.core.paging_lister: Adding comments
  • swh.lister.gitlab: Add the presence check for the incremental lister
  • swh.lister.gitlab.tests: Separate properly tests per lister
  • swh.lister.gitlab: Read next page from headers
  • swh.lister.gitlab: Remove indexable column from gitlab lister
  • swh.lister.gitlab: Change uid format
  • swh.lister.gitlab: Fix the total pages reading instruction
  • swh.lister.gitlab.tasks: Fix range computations
  • swh.lister.core.paging_lister: Fix page initialization
Harbormaster completed remote builds in B1326: Diff 1186.Jul 17 2018, 2:15 PM
Harbormaster completed remote builds in B1327: Diff 1187.
This revision was not accepted when it landed; it landed in state Needs Review.Jul 17 2018, 2:16 PM
Closed by commit rDLSafcd6997c427: swh.lister.cli: Add a basic cli to deal with create/drop db actions (authored by ardumont). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
78 lines
swh/
lister/
bitbucket/
1 line
9 lines
tests/
6 lines
56 lines
core/
20 lines
38 lines
25 lines
157 lines
47 lines
tests/
38 lines
debian/
4 lines
github/
1 line
9 lines
tests/
6 lines
gitlab/
122 lines
30 lines
65 lines
tests/
1 line
170 lines
38 lines
tests/
51 lines
14 lines

Diff 1179

View Options

README.md

Loading...
View Options

swh/lister/bitbucket/lister.py

Loading...
View Options

swh/lister/bitbucket/tasks.py

Loading...
View Options

swh/lister/bitbucket/tests/test_bb_lister.py

Loading...
View Options

swh/lister/cli.py

Loading...
View Options

swh/lister/core/indexing_lister.py

Loading...
View Options

swh/lister/core/lister_base.py

Loading...
View Options

swh/lister/core/lister_transports.py

Loading...
View Options

swh/lister/core/paging_lister.py

Loading...
View Options

swh/lister/core/tasks.py

Loading...
View Options

swh/lister/core/tests/test_lister.py

Loading...
View Options

swh/lister/debian/lister.py

Loading...
View Options

swh/lister/github/lister.py

Loading...
View Options

swh/lister/github/tasks.py

Loading...
View Options

swh/lister/github/tests/test_gh_lister.py

Loading...
View Options

swh/lister/gitlab/__init__.py

Loading...
View Options

swh/lister/gitlab/lister.py

Loading...
View Options

swh/lister/gitlab/models.py

Loading...
View Options

swh/lister/gitlab/tasks.py

Loading...
View Options

swh/lister/gitlab/tests/__init__.py

Loading...
View Options

swh/lister/gitlab/tests/api_empty_response.json

Loading...
View Options

swh/lister/gitlab/tests/api_response.json

Loading...
View Options

swh/lister/gitlab/tests/test_gitlab_lister.py

Loading...
View Options

swh/lister/tests/__init__.py

Loading...
View Options

swh/lister/tests/test_utils.py

Loading...
View Options

swh/lister/utils.py

Loading...
About · Developer information · Legal