Page MenuHomeSoftware Heritage

Added LaunchpadLister
ClosedPublic

Authored by legau on Apr 7 2020, 5:34 PM.

Details

Summary

Related to T1734

From abandonned D2799

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

legau created this revision.Apr 7 2020, 5:34 PM
legau added a comment.Apr 7 2020, 5:37 PM

@ardumont It seems the build error did not go away

@ardumont It seems the build error did not go away

/me grunts
indeed.

meanwhile:

cd swh-lister
git checkout master
git pull
arc patch D2974
tox -r -- -x -s

Everything is fine.

----------- coverage: platform linux, python 3.7.3-final-0 -----------
Name                                                                                   Stmts   Miss Branch BrPart  Cover
------------------------------------------------------------------------------------------------------------------------
.tox/py3/lib/python3.7/site-packages/swh/lister/__init__.py                               20      2      6      1    88%
.tox/py3/lib/python3.7/site-packages/swh/lister/bitbucket/__init__.py                      4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/bitbucket/lister.py                       44      1     10      1    96%
.tox/py3/lib/python3.7/site-packages/swh/lister/bitbucket/models.py                        6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/bitbucket/tasks.py                        27      4      4      1    84%
.tox/py3/lib/python3.7/site-packages/swh/lister/bitbucket/tests/__init__.py                0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/bitbucket/tests/conftest.py                1      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/bitbucket/tests/test_lister.py            62      0      6      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/bitbucket/tests/test_tasks.py             51      0      8      1    98%
.tox/py3/lib/python3.7/site-packages/swh/lister/cgit/__init__.py                           4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cgit/lister.py                            67      2     22      5    92%
.tox/py3/lib/python3.7/site-packages/swh/lister/cgit/models.py                             6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cgit/tasks.py                              6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cgit/tests/__init__.py                     0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cgit/tests/conftest.py                     1      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cgit/tests/test_lister.py                 45      0      4      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cgit/tests/test_tasks.py                  17      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cli.py                                    58     35     16      1    32%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/__init__.py                           0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/abstractattribute.py                  5      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/indexing_lister.py                  105     15     40     10    83%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/lister_base.py                      180     29     54     11    81%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/lister_transports.py                103     28     18      4    72%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/models.py                            35      0      2      1    97%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/page_by_page_lister.py               49     18     14      4    56%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/simple_lister.py                     38      4     10      2    88%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/tests/__init__.py                     0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/tests/conftest.py                    20      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/tests/test_abstractattribute.py      41      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/tests/test_indexing_lister.py        62      1      6      0    99%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/tests/test_lister.py                247      4     38      3    97%
.tox/py3/lib/python3.7/site-packages/swh/lister/core/tests/test_model.py                  56      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cran/__init__.py                           4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cran/lister.py                            41      4      2      1    88%
.tox/py3/lib/python3.7/site-packages/swh/lister/cran/models.py                             6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cran/tasks.py                              6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cran/tests/__init__.py                     0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cran/tests/conftest.py                     6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cran/tests/test_lister.py                 40      0      4      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/cran/tests/test_tasks.py                  17      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/debian/__init__.py                        27      0     10      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/debian/lister.py                         111      7     32      7    90%
.tox/py3/lib/python3.7/site-packages/swh/lister/debian/models.py                         105      5     18      2    94%
.tox/py3/lib/python3.7/site-packages/swh/lister/debian/tasks.py                            6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/debian/tests/__init__.py                   0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/debian/tests/conftest.py                  28      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/debian/tests/test_init.py                 39      0      8      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/debian/tests/test_lister.py               16      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/debian/tests/test_models.py               16      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/debian/tests/test_tasks.py                16      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/debian/utils.py                           43     43     12      0     0%
.tox/py3/lib/python3.7/site-packages/swh/lister/github/__init__.py                         4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/github/lister.py                          38      0      8      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/github/models.py                           7      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/github/tasks.py                           27      4      4      1    84%
.tox/py3/lib/python3.7/site-packages/swh/lister/github/tests/__init__.py                   0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/github/tests/conftest.py                   1      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/github/tests/test_lister.py               43      0      4      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/github/tests/test_tasks.py                51      1      8      2    95%
.tox/py3/lib/python3.7/site-packages/swh/lister/gitlab/__init__.py                         4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gitlab/lister.py                          45     11     12      3    68%
.tox/py3/lib/python3.7/site-packages/swh/lister/gitlab/models.py                           6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gitlab/tasks.py                           27      2      2      0    93%
.tox/py3/lib/python3.7/site-packages/swh/lister/gitlab/tests/__init__.py                   0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gitlab/tests/conftest.py                   1      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gitlab/tests/test_lister.py               42     10      4      0    74%
.tox/py3/lib/python3.7/site-packages/swh/lister/gitlab/tests/test_tasks.py                79      2     16      4    94%
.tox/py3/lib/python3.7/site-packages/swh/lister/gnu/__init__.py                            4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gnu/lister.py                             25      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gnu/models.py                              6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gnu/tasks.py                               6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gnu/tests/__init__.py                      0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gnu/tests/conftest.py                      1      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gnu/tests/test_lister.py                  28      0      6      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gnu/tests/test_tasks.py                   17      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gnu/tests/test_tree.py                    63      0      8      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/gnu/tree.py                               93      0     40      1    99%
.tox/py3/lib/python3.7/site-packages/swh/lister/launchpad/__init__.py                      4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/launchpad/lister.py                       66     13     20      5    74%
.tox/py3/lib/python3.7/site-packages/swh/lister/launchpad/models.py                        6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/launchpad/tasks.py                        13      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/launchpad/tests/__init__.py                0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/launchpad/tests/conftest.py               32      0      8      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/launchpad/tests/test_lister.py            16      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/launchpad/tests/test_tasks.py             21      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/npm/__init__.py                            4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/npm/lister.py                             63     16     12      0    68%
.tox/py3/lib/python3.7/site-packages/swh/lister/npm/models.py                             18      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/npm/tasks.py                              33     15      4      0    49%
.tox/py3/lib/python3.7/site-packages/swh/lister/npm/tests/conftest.py                      6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/npm/tests/test_lister.py                  54      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/npm/tests/test_tasks.py                   35      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/packagist/__init__.py                      4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/packagist/lister.py                       30      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/packagist/models.py                        5      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/packagist/tasks.py                         6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/packagist/tests/__init__.py                0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/packagist/tests/conftest.py                6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/packagist/tests/test_lister.py            49      0      6      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/packagist/tests/test_tasks.py             17      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/phabricator/__init__.py                    4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/phabricator/lister.py                     86      5     36      8    89%
.tox/py3/lib/python3.7/site-packages/swh/lister/phabricator/models.py                      7      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/phabricator/tasks.py                       6      1      0      0    83%
.tox/py3/lib/python3.7/site-packages/swh/lister/phabricator/tests/__init__.py              0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/phabricator/tests/conftest.py              6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/phabricator/tests/test_lister.py          71      0     10      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/phabricator/tests/test_tasks.py            6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/pypi/__init__.py                           4      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/pypi/lister.py                            30      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/pypi/models.py                             5      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/pypi/tasks.py                              6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/pypi/tests/__init__.py                     0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/pypi/tests/conftest.py                     6      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/pypi/tests/test_lister.py                 14      0      2      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/pypi/tests/test_tasks.py                  17      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/tests/__init__.py                          0      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/tests/test_cli.py                         27      0      6      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/tests/test_utils.py                       18      0      0      0   100%
.tox/py3/lib/python3.7/site-packages/swh/lister/utils.py                                   8      0      6      1    93%
------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                   3314    282    580     80    89%

================================================================================================== 125 passed, 1 warning in 98.68s (0:01:38) ==================================================================================================
___________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
  flake8: commands succeeded
  mypy: commands succeeded
  py3: commands succeeded
  congratulations :)

Remains mostly nitpicks to fix (type, indentation, docstring updates...)

swh/lister/launchpad/lister.py
63

...checks: bool = False...

65

indentation is kinda off by 1 char.

(Hopefully we'll have black soon doing that for us),

70

you can remove the type from the docstring.

85

-> Dict[str, Any]:

95

That's returning a dict with the listing status.

swh/lister/launchpad/tests/conftest.py
36

add some space for visibility.

37

same here.

Also, add yourself to the CONTRIBUTORS file (at the top of the repository ;)
(I think it's alphabetic order on the firstname there.)

ardumont added inline comments.Apr 8 2020, 12:42 PM
swh/lister/launchpad/models.py
2

-2020 really (also in other modules ;)

swh/lister/launchpad/tests/data/response0.json
108

Maybe also reduce a bit that file, no need for so many entries.

legau added inline comments.Apr 8 2020, 4:33 PM
swh/lister/launchpad/lister.py
65

What do you mean by that ?

ardumont added inline comments.Apr 8 2020, 4:40 PM
swh/lister/launchpad/lister.py
65
The
 filter

i would expect:

The
filter

or let this as is for now.

someone is blackifying our repositories, so i'd expect this to be dealt with on your next rebase.

legau updated this revision to Diff 10597.Apr 8 2020, 4:43 PM

Adjustments

ardumont accepted this revision.EditedApr 8 2020, 5:25 PM

Ok, so looks good to me.

This needs some last adjustments due to the new black behavior and we are good to go.

Please:

  • Squash your 2 commits into 1.
  • update and rebase on latest master (you should not have conflicts as it's all new files ;)
  • let black do its blackify stuff : pre-commit run --all-files (should trigger black on your diff)
  • and then amend your commit and update this diff

Thanks in advance.

Cheers,

This revision is now accepted and ready to land.Apr 8 2020, 5:25 PM
ardumont edited the summary of this revision. (Show Details)Apr 9 2020, 9:59 AM
legau updated this revision to Diff 10705.Apr 12 2020, 12:55 AM

Merged with master

Build is green

Patch application report for D2974 (id=10705)

Rebasing onto 29325c1d94...

Current branch diff-target is up to date.
Changes applied before test
commit 1fff5ae2be436666cdb58fdf1942f5f66557febd
Author: Léni Gauffier <lenigauffier@gmail.com>
Date:   Sun Apr 12 00:22:20 2020 +0200

    Added LaunchpadLister

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/9/ for more details.

This revision was automatically updated to reflect the committed changes.

\m/

Thanks!

Note:

  • your commit in master worked (as expected)
  • For some reason this diff build finally passed! ¯\_(ツ)_/¯

Cheers,