Page MenuHomeSoftware Heritage

Bootstrap gitlab lister
ClosedPublic

Authored by ardumont on Jul 3 2018, 12:42 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Commits
rDLSd52089154775: swh.lister.core.paging_lister: Adding comments
rDLS847a8d341a8b: swh.lister.gitlab: Add Incremental lister behavior
rDLSccd0525c9b5f: swh.lister: Do not hardcode the index notion into parameter names
rDLS4c4aa0ead2c1: swh.lister: Make LISTER_NAME a class attribute
rDLSb6c5865ab17d: swh.lister.paging_lister: Improve lister's base class name
rDLSa51c36194e4a: swh.lister.gitlab: Add full gitlab lister
rDLS581028cfc5a6: swh.lister.cli: Fix cli docstring
rDLS7954e03627d3: swh.lister: Document swh.lister.tasks's intent
rDLSba146376d66e: swh.lister: Add tests around the gitlab lister
rDLSe1a460caa5d2: swh.lister.gitlab: Improve docstring
rDLS1fd44207eb44: README: Add a run section to demonstrate how to run a github lister
rDLS79cd00737f80: swh.lister.gitlab: Remove TODO about the 403 response code
rDLS3ca566776fe2: swh.lister.gitlab: Make rate limit check optional
rDLSdb36c499fe97: swh.lister.gitlab: Do not store information we cannot have
rDLS935b9cd24f45: swh.lister.core: Make gitlab lister a paging lister instance
rDLS9c5963f0155e: swh.lister.gitlab: Respect camel case convention
rDLS3760708897fc: swh.lister.gitlab: Use one configuration for the gitlab lister
rDLSd9d582f7ebdd: swh.lister.gitlab: Change the default sort order using id
rDLS8ad70b3d60ea: swh.lister.gitlab: Make the lister instance parametric
rDLSf8055a34afc9: swh.lister.gitlab: Add notes around current limitations
rDLS7a0e3a237964: gitlab.lister: Override filtering as it's not possible as is
rDLS1335a5b25f69: swh.lister.gitlab: Use gitlab's index identifier (page id)
rDLS0e517b614334: swh.lister.gitlab: Use http_url_to_repo key to build the origin url
rDLS3e62bc867ea4: swh.lister.cli: Simplify cli
rDLSafcd6997c427: swh.lister.cli: Add a basic cli to deal with create/drop db actions
Summary

Nothing changes between D351 and this except for:

  • separating the documentation cleanup patches
  • D351 got closed wrongly

There still remains to take into account the good remarks D351#6877
and D351#6882.

835c605 * lister-gitlab swh.lister.gitlab: Add notes around current limitations
15d3feb * swh.lister.gitlab: Use http_url_to_repo key to build the origin url
0528603 * gitlab.lister: Override filtering as it's not possible as is
f5b35a9 * swh.lister.gitlab: Use gitlab's index identifier (page id)
49439ab * swh.lister.cli: Add a basic cli to deal with create/drop db actions
0f9ba7b * swh.lister.gitlab: Bootstrap gitlab lister

Related D351
Related T989

Test Plan

To be added

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Update:

  • swh.lister.gitlab: Respect camel case convention
  • swh.lister.gitlab: Make the lister instance parametric
  • swh.lister.gitlab: Make the lister's task instance parametric
  • swh.lister.gitlab: Change the default sort order using id
  • swh.lister.gitlab: Do not store information we cannot have

Updates:

  • swh.lister.gitlab: Change the default sort order using id
  • swh.lister.gitlab: Do not store information we cannot have
  • swh.lister.core: Make gitlab lister a paging lister instance

TODO: Add tests

Updates:

  • swh.lister.core: Make gitlab lister a paging lister instance
  • README: Add a run section to demonstrate how to run a github lister
  • swh.lister.gitlab: Remove TODO about the 403 response code
  • swh.lister.gitlab: Use one configuration for the gitlab lister
  • swh.lister.gitlab: Make rate limit check optional
  • README.md: Add a section to demo how to run the gitlab lister
  • swh.lister.gitlab: Improve docstring
  • swh.lister: Add tests around the gitlab lister
  • swh.lister: Document swh.lister.tasks's intent

Manual tests:

Remains to do:

  • Add incremental/refresh policy for the gitlab lister
  • Properly separate the indexing/paging tests (cf. FIXME)
  • swh.lister.gitlab: Add full gitlab lister

Clean up gitlab lister part not related to the bootstrap

Rebase (and adapt) according to review from D356 to D376:

  • swh.lister.cli: Simplify cli
  • swh.lister.gitlab: Use gitlab's index identifier (page id)
  • gitlab.lister: Override filtering as it's not possible as is
  • swh.lister.gitlab: Use http_url_to_repo key to build the origin url
  • swh.lister.gitlab: Add notes around current limitations
  • swh.lister.gitlab: Respect camel case convention
  • swh.lister.gitlab: Make the lister instance parametric
  • swh.lister.gitlab: Use one configuration for the gitlab lister
  • swh.lister.gitlab: Make the lister's task instance parametric
  • swh.lister.gitlab: Change the default sort order using id
  • swh.lister.gitlab: Do not store information we cannot have
  • swh.lister.core: Make gitlab lister a paging lister instance
  • README: Add a run section to demonstrate how to run a github lister
  • swh.lister.gitlab: Remove TODO about the 403 response code
  • swh.lister.gitlab: Make rate limit check optional
  • README.md: Add a section to demo how to run the gitlab lister
  • swh.lister.gitlab: Improve docstring
  • swh.lister: Add tests around the gitlab lister
  • swh.lister: Document swh.lister.tasks's intent
  • swh.lister.gitlab: Add full gitlab lister
  • swh.lister.cli: Fix cli docstring
  • swh.lister: Make LISTER_NAME a class attribute
  • swh.lister.paging_lister: Improve lister's base class name
  • swh.lister: Do not hardcode the index notion into parameter names
  • swh.lister.gitlab: Add Incremental lister behavior
  • swh.lister.gitlab: Improve headers extraction
  • swh.lister.gitlab.tasks: Remove spurious comma
  • swh.lister.core.paging_lister: Adding comments
  • swh.lister.gitlab: Add the presence check for the incremental lister
  • swh.lister.gitlab.tests: Separate properly tests per lister
  • swh.lister.gitlab: Read next page from headers

Fix multiple issues

  • swh.lister.gitlab: Remove indexable column from gitlab lister
  • swh.lister.gitlab: Change uid format
  • swh.lister.gitlab: Fix the total pages reading instruction
  • swh.lister.gitlab.tasks: Fix range computations
  • swh.lister.core.paging_lister: Fix page initialization

Tests are fine for:

  • range lister
  • full lister
  • incremental lister

swh.lister.gitlab: Add Incremental lister behavior

Made a mistake in the arc runes...

(forgot that once a diff is opened for a branch, you must specify that
you want to create a new one...)

  • swh.lister.gitlab: Improve headers extraction
  • swh.lister.gitlab.tasks: Remove spurious comma
  • swh.lister.core.paging_lister: Adding comments
  • swh.lister.gitlab: Add the presence check for the incremental lister
  • swh.lister.gitlab.tests: Separate properly tests per lister
  • swh.lister.gitlab: Read next page from headers
  • swh.lister.gitlab: Remove indexable column from gitlab lister
  • swh.lister.gitlab: Change uid format
  • swh.lister.gitlab: Fix the total pages reading instruction
  • swh.lister.gitlab.tasks: Fix range computations
  • swh.lister.core.paging_lister: Fix page initialization
This revision was not accepted when it landed; it landed in state Needs Review.Jul 17 2018, 2:16 PM
This revision was automatically updated to reflect the committed changes.