Differential D4992
Remove no longer used legacy Lister API and update CLI options Authored by anlambert on Feb 2 2021, 12:45 PM.
Details
Legacy Lister classes from the swh.lister.core mdule are no longer Also remove lister CLI options related to legacy Lister API. As a consequence, the following requirements are no longer needed: Closes T2442 Depends on D4990
Diff Detail
Event TimelineComment Actions Build is green Patch application report for D4992 (id=17801)Could not rebase; Attempt merge onto 8e4dd178f1... Updating 8e4dd17..0abff44 Fast-forward conftest.py | 4 +- mypy.ini | 4 - requirements-test.txt | 2 - requirements.txt | 2 - swh/lister/cli.py | 92 +--- swh/lister/core/__init__.py | 0 swh/lister/core/abstractattribute.py | 28 -- swh/lister/core/lister_base.py | 508 --------------------- swh/lister/core/lister_transports.py | 233 ---------- swh/lister/core/models.py | 77 ---- swh/lister/core/simple_lister.py | 96 ---- swh/lister/core/tests/__init__.py | 0 swh/lister/core/tests/test_abstractattribute.py | 64 --- swh/lister/core/tests/test_lister.py | 453 ------------------ swh/lister/core/tests/test_model.py | 91 ---- swh/lister/packagist/__init__.py | 4 +- swh/lister/packagist/lister.py | 230 +++++++--- swh/lister/packagist/models.py | 17 - swh/lister/packagist/tasks.py | 4 +- swh/lister/packagist/tests/conftest.py | 26 -- .../data/https_packagist.org/packages_list.json | 9 - .../packagist/tests/data/ljjackson_linnworks.json | 83 ++++ .../packagist/tests/data/lky_wx_article.json | 240 ++++++++++ .../tests/data/spryker-eco_computop-api.json | 145 ++++++ swh/lister/packagist/tests/test_lister.py | 164 +++---- swh/lister/packagist/tests/test_tasks.py | 18 +- swh/lister/pytest_plugin.py | 62 --- swh/lister/tests/test_cli.py | 6 +- swh/lister/tests/test_utils.py | 13 - 29 files changed, 726 insertions(+), 1949 deletions(-) delete mode 100644 swh/lister/core/__init__.py delete mode 100644 swh/lister/core/abstractattribute.py delete mode 100644 swh/lister/core/lister_base.py delete mode 100644 swh/lister/core/lister_transports.py delete mode 100644 swh/lister/core/models.py delete mode 100644 swh/lister/core/simple_lister.py delete mode 100644 swh/lister/core/tests/__init__.py delete mode 100644 swh/lister/core/tests/test_abstractattribute.py delete mode 100644 swh/lister/core/tests/test_lister.py delete mode 100644 swh/lister/core/tests/test_model.py delete mode 100644 swh/lister/packagist/models.py delete mode 100644 swh/lister/packagist/tests/conftest.py delete mode 100644 swh/lister/packagist/tests/data/https_packagist.org/packages_list.json create mode 100644 swh/lister/packagist/tests/data/ljjackson_linnworks.json create mode 100644 swh/lister/packagist/tests/data/lky_wx_article.json create mode 100644 swh/lister/packagist/tests/data/spryker-eco_computop-api.json delete mode 100644 swh/lister/pytest_plugin.py Changes applied before testcommit 0abff440a28abf8b23a536bf38fb2e68ce87a467
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date: Tue Feb 2 12:27:46 2021 +0100
Remove no longer used legacy Lister API and update CLI options
Legacy Lister classes from the swh.lister.core mdule are no longer
used in swh-lister codebase so it is time to remove them.
Also remove lister CLI options related to legacy Lister API.
As a consequence, the following requirements are no longer needed:
arrow, SQLAlchemy, sqlalchemy-stubs and testing.postgresql.
Closes T2442
commit 478081c1513b240f85c78cc66e9a3109eff91608
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date: Mon Feb 1 17:34:10 2021 +0100
packagist: Reimplement lister using new Lister API
The previous implementation was generating tasks for a non implemented
Packagist loader.
The new implementation extracts source repository URL, VCS type and
last update date for each package referenced by Packagist and send
those info to the scheduler.
Packages metadata are retrieved using Packagist API endpoints whose
responses are served from static files, which are guaranteed to be
efficient on the Packagist side (no dymamic queries).
Furthermore, subsequent listing will send the "If-Modified-Since" HTTP
header to only retrieve packages metadata updated since the previous
listing operation in order to save bandwidth and return only origins
which might have new released versions.
Closes T2991See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/236/ for more details. Comment Actions Build is green Patch application report for D4992 (id=17826)Could not rebase; Attempt merge onto 82ab96ad06... Updating 82ab96a..8933544 Fast-forward conftest.py | 4 +- mypy.ini | 7 - requirements-test.txt | 2 - requirements.txt | 2 - swh/lister/cli.py | 92 +--- swh/lister/core/__init__.py | 0 swh/lister/core/abstractattribute.py | 28 -- swh/lister/core/lister_base.py | 508 --------------------- swh/lister/core/lister_transports.py | 233 ---------- swh/lister/core/models.py | 77 ---- swh/lister/core/simple_lister.py | 96 ---- swh/lister/core/tests/__init__.py | 0 swh/lister/core/tests/test_abstractattribute.py | 64 --- swh/lister/core/tests/test_lister.py | 453 ------------------ swh/lister/core/tests/test_model.py | 91 ---- swh/lister/packagist/__init__.py | 4 +- swh/lister/packagist/lister.py | 228 ++++++--- swh/lister/packagist/models.py | 17 - swh/lister/packagist/tasks.py | 4 +- swh/lister/packagist/tests/conftest.py | 26 -- .../packagist/tests/data/den1n_contextmenu.json | 78 ++++ .../data/https_packagist.org/packages_list.json | 9 - .../packagist/tests/data/ljjackson_linnworks.json | 83 ++++ .../packagist/tests/data/lky_wx_article.json | 239 ++++++++++ .../tests/data/spryker-eco_computop-api.json | 141 ++++++ swh/lister/packagist/tests/test_lister.py | 211 +++++---- swh/lister/packagist/tests/test_tasks.py | 18 +- swh/lister/pytest_plugin.py | 62 --- swh/lister/tests/test_cli.py | 6 +- swh/lister/tests/test_utils.py | 13 - 30 files changed, 848 insertions(+), 1948 deletions(-) delete mode 100644 swh/lister/core/__init__.py delete mode 100644 swh/lister/core/abstractattribute.py delete mode 100644 swh/lister/core/lister_base.py delete mode 100644 swh/lister/core/lister_transports.py delete mode 100644 swh/lister/core/models.py delete mode 100644 swh/lister/core/simple_lister.py delete mode 100644 swh/lister/core/tests/__init__.py delete mode 100644 swh/lister/core/tests/test_abstractattribute.py delete mode 100644 swh/lister/core/tests/test_lister.py delete mode 100644 swh/lister/core/tests/test_model.py delete mode 100644 swh/lister/packagist/models.py delete mode 100644 swh/lister/packagist/tests/conftest.py create mode 100644 swh/lister/packagist/tests/data/den1n_contextmenu.json delete mode 100644 swh/lister/packagist/tests/data/https_packagist.org/packages_list.json create mode 100644 swh/lister/packagist/tests/data/ljjackson_linnworks.json create mode 100644 swh/lister/packagist/tests/data/lky_wx_article.json create mode 100644 swh/lister/packagist/tests/data/spryker-eco_computop-api.json delete mode 100644 swh/lister/pytest_plugin.py Changes applied before testcommit 89335445210fc0ce6ce9eab937741d166919a92e
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date: Tue Feb 2 12:27:46 2021 +0100
Remove no longer used legacy Lister API and update CLI options
Legacy Lister classes from the swh.lister.core mdule are no longer
used in swh-lister codebase so it is time to remove them.
Also remove lister CLI options related to legacy Lister API.
As a consequence, the following requirements are no longer needed:
arrow, SQLAlchemy, sqlalchemy-stubs and testing.postgresql.
Closes T2442
commit ff05191b7db7b217c8682e9888338b8813e2df6a
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date: Mon Feb 1 17:34:10 2021 +0100
packagist: Reimplement lister using new Lister API
The previous implementation was generating tasks for a non implemented
Packagist loader.
The new implementation extracts source repository URL, VCS type and
last update date for each package referenced by Packagist and send
those info to the scheduler.
Packages metadata are retrieved using Packagist API endpoints whose
responses are served from static files, which are guaranteed to be
efficient on the Packagist side (no dymamic queries).
Furthermore, subsequent listing will send the "If-Modified-Since" HTTP
header to only retrieve packages metadata updated since the previous
listing operation in order to save bandwidth and return only origins
which might have new released versions.
Closes T2991See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/239/ for more details. |