Page MenuHomeSoftware Heritage
Differential D352

Bootstrap gitlab lister
ClosedPublic
Actions

Authored by ardumont on Jul 3 2018, 12:42 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Commits
rDLSd52089154775: swh.lister.core.paging_lister: Adding comments
rDLS847a8d341a8b: swh.lister.gitlab: Add Incremental lister behavior
rDLSccd0525c9b5f: swh.lister: Do not hardcode the index notion into parameter names
rDLS4c4aa0ead2c1: swh.lister: Make LISTER_NAME a class attribute
rDLSb6c5865ab17d: swh.lister.paging_lister: Improve lister's base class name
rDLSa51c36194e4a: swh.lister.gitlab: Add full gitlab lister
rDLS581028cfc5a6: swh.lister.cli: Fix cli docstring
rDLS7954e03627d3: swh.lister: Document swh.lister.tasks's intent
rDLSba146376d66e: swh.lister: Add tests around the gitlab lister
rDLSe1a460caa5d2: swh.lister.gitlab: Improve docstring
rDLS1fd44207eb44: README: Add a run section to demonstrate how to run a github lister
rDLS79cd00737f80: swh.lister.gitlab: Remove TODO about the 403 response code
rDLS3ca566776fe2: swh.lister.gitlab: Make rate limit check optional
rDLSdb36c499fe97: swh.lister.gitlab: Do not store information we cannot have
rDLS935b9cd24f45: swh.lister.core: Make gitlab lister a paging lister instance
rDLS9c5963f0155e: swh.lister.gitlab: Respect camel case convention
rDLS3760708897fc: swh.lister.gitlab: Use one configuration for the gitlab lister
rDLSd9d582f7ebdd: swh.lister.gitlab: Change the default sort order using id
rDLS8ad70b3d60ea: swh.lister.gitlab: Make the lister instance parametric
rDLSf8055a34afc9: swh.lister.gitlab: Add notes around current limitations
rDLS7a0e3a237964: gitlab.lister: Override filtering as it's not possible as is
rDLS1335a5b25f69: swh.lister.gitlab: Use gitlab's index identifier (page id)
rDLS0e517b614334: swh.lister.gitlab: Use http_url_to_repo key to build the origin url
rDLS3e62bc867ea4: swh.lister.cli: Simplify cli
rDLSafcd6997c427: swh.lister.cli: Add a basic cli to deal with create/drop db actions
Summary

Nothing changes between D351 and this except for:

  • separating the documentation cleanup patches
  • D351 got closed wrongly

There still remains to take into account the good remarks D351#6877
and D351#6882.

835c605 * lister-gitlab swh.lister.gitlab: Add notes around current limitations
15d3feb * swh.lister.gitlab: Use http_url_to_repo key to build the origin url
0528603 * gitlab.lister: Override filtering as it's not possible as is
f5b35a9 * swh.lister.gitlab: Use gitlab's index identifier (page id)
49439ab * swh.lister.cli: Add a basic cli to deal with create/drop db actions
0f9ba7b * swh.lister.gitlab: Bootstrap gitlab lister

Related D351
Related T989

Test Plan

To be added

Diff Detail

Repository
rDLS Listers
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 1326
Build 1670: arc lint + arc unit

Event Timeline

ardumont created this revision.Jul 3 2018, 12:42 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 3 2018, 12:42 PM
Harbormaster completed remote builds in B1268: Diff 1126.Jul 3 2018, 12:43 PM
ardumont updated this revision to Diff 1127.Jul 3 2018, 4:33 PM

Update:

  • swh.lister.gitlab: Respect camel case convention
  • swh.lister.gitlab: Make the lister instance parametric
  • swh.lister.gitlab: Make the lister's task instance parametric
  • swh.lister.gitlab: Change the default sort order using id
  • swh.lister.gitlab: Do not store information we cannot have
Harbormaster completed remote builds in B1269: Diff 1127.Jul 3 2018, 4:33 PM
ardumont updated this revision to Diff 1128.Jul 3 2018, 6:09 PM

Updates:

  • swh.lister.gitlab: Change the default sort order using id
  • swh.lister.gitlab: Do not store information we cannot have
  • swh.lister.core: Make gitlab lister a paging lister instance

TODO: Add tests

Harbormaster completed remote builds in B1270: Diff 1128.Jul 3 2018, 6:09 PM
ardumont updated this revision to Diff 1129.Jul 4 2018, 4:38 PM

Updates:

  • swh.lister.core: Make gitlab lister a paging lister instance
  • README: Add a run section to demonstrate how to run a github lister
  • swh.lister.gitlab: Remove TODO about the 403 response code
  • swh.lister.gitlab: Use one configuration for the gitlab lister
  • swh.lister.gitlab: Make rate limit check optional
  • README.md: Add a section to demo how to run the gitlab lister
  • swh.lister.gitlab: Improve docstring
  • swh.lister: Add tests around the gitlab lister
  • swh.lister: Document swh.lister.tasks's intent

Manual tests:

Remains to do:

  • Add incremental/refresh policy for the gitlab lister
  • Properly separate the indexing/paging tests (cf. FIXME)
Harbormaster completed remote builds in B1271: Diff 1129.Jul 4 2018, 4:38 PM
ardumont updated this revision to Diff 1130.Jul 4 2018, 6:58 PM
  • swh.lister.gitlab: Add full gitlab lister
Harbormaster completed remote builds in B1272: Diff 1130.Jul 4 2018, 6:58 PM
ardumont updated this revision to Diff 1136.Jul 6 2018, 3:55 PM

Clean up gitlab lister part not related to the bootstrap

Harbormaster completed remote builds in B1277: Diff 1136.Jul 6 2018, 3:55 PM
ardumont mentioned this in D363: swh.lister.gitlab: Make the lister instance parametric.Jul 11 2018, 11:01 AM
ardumont updated this revision to Diff 1179.EditedJul 12 2018, 12:31 PM

Rebase (and adapt) according to review from D356 to D376:

  • swh.lister.cli: Simplify cli
  • swh.lister.gitlab: Use gitlab's index identifier (page id)
  • gitlab.lister: Override filtering as it's not possible as is
  • swh.lister.gitlab: Use http_url_to_repo key to build the origin url
  • swh.lister.gitlab: Add notes around current limitations
  • swh.lister.gitlab: Respect camel case convention
  • swh.lister.gitlab: Make the lister instance parametric
  • swh.lister.gitlab: Use one configuration for the gitlab lister
  • swh.lister.gitlab: Make the lister's task instance parametric
  • swh.lister.gitlab: Change the default sort order using id
  • swh.lister.gitlab: Do not store information we cannot have
  • swh.lister.core: Make gitlab lister a paging lister instance
  • README: Add a run section to demonstrate how to run a github lister
  • swh.lister.gitlab: Remove TODO about the 403 response code
  • swh.lister.gitlab: Make rate limit check optional
  • README.md: Add a section to demo how to run the gitlab lister
  • swh.lister.gitlab: Improve docstring
  • swh.lister: Add tests around the gitlab lister
  • swh.lister: Document swh.lister.tasks's intent
  • swh.lister.gitlab: Add full gitlab lister
  • swh.lister.cli: Fix cli docstring
  • swh.lister: Make LISTER_NAME a class attribute
  • swh.lister.paging_lister: Improve lister's base class name
  • swh.lister: Do not hardcode the index notion into parameter names
  • swh.lister.gitlab: Add Incremental lister behavior
  • swh.lister.gitlab: Improve headers extraction
  • swh.lister.gitlab.tasks: Remove spurious comma
  • swh.lister.core.paging_lister: Adding comments
  • swh.lister.gitlab: Add the presence check for the incremental lister
  • swh.lister.gitlab.tests: Separate properly tests per lister
  • swh.lister.gitlab: Read next page from headers
Harbormaster completed remote builds in B1319: Diff 1179.Jul 12 2018, 12:31 PM
ardumont updated this revision to Diff 1180.Jul 12 2018, 3:20 PM

Fix multiple issues

  • swh.lister.gitlab: Remove indexable column from gitlab lister
  • swh.lister.gitlab: Change uid format
  • swh.lister.gitlab: Fix the total pages reading instruction
  • swh.lister.gitlab.tasks: Fix range computations
  • swh.lister.core.paging_lister: Fix page initialization

Tests are fine for:

  • range lister
  • full lister
  • incremental lister
Harbormaster completed remote builds in B1320: Diff 1180.Jul 12 2018, 3:20 PM
ardumont updated this revision to Diff 1186.Jul 16 2018, 9:50 AM

swh.lister.gitlab: Add Incremental lister behavior

ardumont updated this revision to Diff 1187.Jul 16 2018, 9:53 AM

Made a mistake in the arc runes...

(forgot that once a diff is opened for a branch, you must specify that
you want to create a new one...)

  • swh.lister.gitlab: Improve headers extraction
  • swh.lister.gitlab.tasks: Remove spurious comma
  • swh.lister.core.paging_lister: Adding comments
  • swh.lister.gitlab: Add the presence check for the incremental lister
  • swh.lister.gitlab.tests: Separate properly tests per lister
  • swh.lister.gitlab: Read next page from headers
  • swh.lister.gitlab: Remove indexable column from gitlab lister
  • swh.lister.gitlab: Change uid format
  • swh.lister.gitlab: Fix the total pages reading instruction
  • swh.lister.gitlab.tasks: Fix range computations
  • swh.lister.core.paging_lister: Fix page initialization
Harbormaster completed remote builds in B1326: Diff 1186.Jul 17 2018, 2:15 PM
Harbormaster completed remote builds in B1327: Diff 1187.
This revision was not accepted when it landed; it landed in state Needs Review.Jul 17 2018, 2:16 PM
Closed by commit rDLSafcd6997c427: swh.lister.cli: Add a basic cli to deal with create/drop db actions (authored by ardumont). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
swh/
lister/
core/
19 lines
gitlab/
4 lines
24 lines

Diff 1186

View Options

swh/lister/core/paging_lister.py

Loading...
View Options

swh/lister/gitlab/lister.py

Loading...
View Options

swh/lister/gitlab/tasks.py

Loading...
About · Developer information · Legal