Page MenuHomeSoftware Heritage

Remove no longer used legacy Lister API and update CLI options
ClosedPublic

Authored by anlambert on Feb 2 2021, 12:45 PM.

Details

Summary

Legacy Lister classes from the swh.lister.core mdule are no longer
used in swh-lister codebase so it is time to remove them.

Also remove lister CLI options related to legacy Lister API.

As a consequence, the following requirements are no longer needed:
arrow, SQLAlchemy, sqlalchemy-stubs and testing.postgresql.

Closes T2442

Depends on D4990

Diff Detail

Repository
rDLS Listers
Branch
lister-remove-core
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18942
Build 29355: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 29354: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D4992 (id=17801)

Could not rebase; Attempt merge onto 8e4dd178f1...

Updating 8e4dd17..0abff44
Fast-forward
 conftest.py                                        |   4 +-
 mypy.ini                                           |   4 -
 requirements-test.txt                              |   2 -
 requirements.txt                                   |   2 -
 swh/lister/cli.py                                  |  92 +---
 swh/lister/core/__init__.py                        |   0
 swh/lister/core/abstractattribute.py               |  28 --
 swh/lister/core/lister_base.py                     | 508 ---------------------
 swh/lister/core/lister_transports.py               | 233 ----------
 swh/lister/core/models.py                          |  77 ----
 swh/lister/core/simple_lister.py                   |  96 ----
 swh/lister/core/tests/__init__.py                  |   0
 swh/lister/core/tests/test_abstractattribute.py    |  64 ---
 swh/lister/core/tests/test_lister.py               | 453 ------------------
 swh/lister/core/tests/test_model.py                |  91 ----
 swh/lister/packagist/__init__.py                   |   4 +-
 swh/lister/packagist/lister.py                     | 230 +++++++---
 swh/lister/packagist/models.py                     |  17 -
 swh/lister/packagist/tasks.py                      |   4 +-
 swh/lister/packagist/tests/conftest.py             |  26 --
 .../data/https_packagist.org/packages_list.json    |   9 -
 .../packagist/tests/data/ljjackson_linnworks.json  |  83 ++++
 .../packagist/tests/data/lky_wx_article.json       | 240 ++++++++++
 .../tests/data/spryker-eco_computop-api.json       | 145 ++++++
 swh/lister/packagist/tests/test_lister.py          | 164 +++----
 swh/lister/packagist/tests/test_tasks.py           |  18 +-
 swh/lister/pytest_plugin.py                        |  62 ---
 swh/lister/tests/test_cli.py                       |   6 +-
 swh/lister/tests/test_utils.py                     |  13 -
 29 files changed, 726 insertions(+), 1949 deletions(-)
 delete mode 100644 swh/lister/core/__init__.py
 delete mode 100644 swh/lister/core/abstractattribute.py
 delete mode 100644 swh/lister/core/lister_base.py
 delete mode 100644 swh/lister/core/lister_transports.py
 delete mode 100644 swh/lister/core/models.py
 delete mode 100644 swh/lister/core/simple_lister.py
 delete mode 100644 swh/lister/core/tests/__init__.py
 delete mode 100644 swh/lister/core/tests/test_abstractattribute.py
 delete mode 100644 swh/lister/core/tests/test_lister.py
 delete mode 100644 swh/lister/core/tests/test_model.py
 delete mode 100644 swh/lister/packagist/models.py
 delete mode 100644 swh/lister/packagist/tests/conftest.py
 delete mode 100644 swh/lister/packagist/tests/data/https_packagist.org/packages_list.json
 create mode 100644 swh/lister/packagist/tests/data/ljjackson_linnworks.json
 create mode 100644 swh/lister/packagist/tests/data/lky_wx_article.json
 create mode 100644 swh/lister/packagist/tests/data/spryker-eco_computop-api.json
 delete mode 100644 swh/lister/pytest_plugin.py
Changes applied before test
commit 0abff440a28abf8b23a536bf38fb2e68ce87a467
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Feb 2 12:27:46 2021 +0100

    Remove no longer used legacy Lister API and update CLI options
    
    Legacy Lister classes from the swh.lister.core mdule are no longer
    used in swh-lister codebase so it is time to remove them.
    
    Also remove lister CLI options related to legacy Lister API.
    
    As a consequence, the following requirements are no longer needed:
    arrow, SQLAlchemy, sqlalchemy-stubs and testing.postgresql.
    
    Closes T2442

commit 478081c1513b240f85c78cc66e9a3109eff91608
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Mon Feb 1 17:34:10 2021 +0100

    packagist: Reimplement lister using new Lister API
    
    The previous implementation was generating tasks for a non implemented
    Packagist loader.
    
    The new implementation extracts source repository URL, VCS type and
    last update date for each package referenced by Packagist and send
    those info to the scheduler.
    
    Packages metadata are retrieved using Packagist API endpoints whose
    responses are served from static files, which are guaranteed to be
    efficient on the Packagist side (no dymamic queries).
    Furthermore, subsequent listing will send the "If-Modified-Since" HTTP
    header to only retrieve packages metadata updated since the previous
    listing operation in order to save bandwidth and return only origins
    which might have new released versions.
    
    Closes T2991

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/236/ for more details.

This revision is now accepted and ready to land.Feb 2 2021, 1:37 PM

Build is green

Patch application report for D4992 (id=17826)

Could not rebase; Attempt merge onto 82ab96ad06...

Updating 82ab96a..8933544
Fast-forward
 conftest.py                                        |   4 +-
 mypy.ini                                           |   7 -
 requirements-test.txt                              |   2 -
 requirements.txt                                   |   2 -
 swh/lister/cli.py                                  |  92 +---
 swh/lister/core/__init__.py                        |   0
 swh/lister/core/abstractattribute.py               |  28 --
 swh/lister/core/lister_base.py                     | 508 ---------------------
 swh/lister/core/lister_transports.py               | 233 ----------
 swh/lister/core/models.py                          |  77 ----
 swh/lister/core/simple_lister.py                   |  96 ----
 swh/lister/core/tests/__init__.py                  |   0
 swh/lister/core/tests/test_abstractattribute.py    |  64 ---
 swh/lister/core/tests/test_lister.py               | 453 ------------------
 swh/lister/core/tests/test_model.py                |  91 ----
 swh/lister/packagist/__init__.py                   |   4 +-
 swh/lister/packagist/lister.py                     | 228 ++++++---
 swh/lister/packagist/models.py                     |  17 -
 swh/lister/packagist/tasks.py                      |   4 +-
 swh/lister/packagist/tests/conftest.py             |  26 --
 .../packagist/tests/data/den1n_contextmenu.json    |  78 ++++
 .../data/https_packagist.org/packages_list.json    |   9 -
 .../packagist/tests/data/ljjackson_linnworks.json  |  83 ++++
 .../packagist/tests/data/lky_wx_article.json       | 239 ++++++++++
 .../tests/data/spryker-eco_computop-api.json       | 141 ++++++
 swh/lister/packagist/tests/test_lister.py          | 211 +++++----
 swh/lister/packagist/tests/test_tasks.py           |  18 +-
 swh/lister/pytest_plugin.py                        |  62 ---
 swh/lister/tests/test_cli.py                       |   6 +-
 swh/lister/tests/test_utils.py                     |  13 -
 30 files changed, 848 insertions(+), 1948 deletions(-)
 delete mode 100644 swh/lister/core/__init__.py
 delete mode 100644 swh/lister/core/abstractattribute.py
 delete mode 100644 swh/lister/core/lister_base.py
 delete mode 100644 swh/lister/core/lister_transports.py
 delete mode 100644 swh/lister/core/models.py
 delete mode 100644 swh/lister/core/simple_lister.py
 delete mode 100644 swh/lister/core/tests/__init__.py
 delete mode 100644 swh/lister/core/tests/test_abstractattribute.py
 delete mode 100644 swh/lister/core/tests/test_lister.py
 delete mode 100644 swh/lister/core/tests/test_model.py
 delete mode 100644 swh/lister/packagist/models.py
 delete mode 100644 swh/lister/packagist/tests/conftest.py
 create mode 100644 swh/lister/packagist/tests/data/den1n_contextmenu.json
 delete mode 100644 swh/lister/packagist/tests/data/https_packagist.org/packages_list.json
 create mode 100644 swh/lister/packagist/tests/data/ljjackson_linnworks.json
 create mode 100644 swh/lister/packagist/tests/data/lky_wx_article.json
 create mode 100644 swh/lister/packagist/tests/data/spryker-eco_computop-api.json
 delete mode 100644 swh/lister/pytest_plugin.py
Changes applied before test
commit 89335445210fc0ce6ce9eab937741d166919a92e
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Feb 2 12:27:46 2021 +0100

    Remove no longer used legacy Lister API and update CLI options
    
    Legacy Lister classes from the swh.lister.core mdule are no longer
    used in swh-lister codebase so it is time to remove them.
    
    Also remove lister CLI options related to legacy Lister API.
    
    As a consequence, the following requirements are no longer needed:
    arrow, SQLAlchemy, sqlalchemy-stubs and testing.postgresql.
    
    Closes T2442

commit ff05191b7db7b217c8682e9888338b8813e2df6a
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Mon Feb 1 17:34:10 2021 +0100

    packagist: Reimplement lister using new Lister API
    
    The previous implementation was generating tasks for a non implemented
    Packagist loader.
    
    The new implementation extracts source repository URL, VCS type and
    last update date for each package referenced by Packagist and send
    those info to the scheduler.
    
    Packages metadata are retrieved using Packagist API endpoints whose
    responses are served from static files, which are guaranteed to be
    efficient on the Packagist side (no dymamic queries).
    Furthermore, subsequent listing will send the "If-Modified-Since" HTTP
    header to only retrieve packages metadata updated since the previous
    listing operation in order to save bandwidth and return only origins
    which might have new released versions.
    
    Closes T2991

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/239/ for more details.