Page MenuHomeSoftware Heritage

Hook up recently introduced options to all listers
ClosedPublic

Authored by olasd on Dec 5 2022, 4:38 PM.

Details

Summary

Hopefully one day we'll be able to replace all of this mess with PEP692
TypedDict kwargs, but that's only on track for Python 3.12.

Test Plan

swh lister run -l gitlab url=https://git.drupalcode.org/api/v4 max_pages=5 max_origins_per_page=15 enable_origins=False 'ignored_project_prefixes=["issues/"]' does something sensible

Diff Detail

Repository
rDLS Listers
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 33087
Build 51865: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 51864: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D8921 (id=32134)

Could not rebase; Attempt merge onto 64267f8f50...

Updating 64267f8..e785e67
Fast-forward
 swh/lister/arch/lister.py              |  6 ++++++
 swh/lister/aur/lister.py               |  6 ++++++
 swh/lister/bitbucket/lister.py         |  6 ++++++
 swh/lister/bower/lister.py             |  6 ++++++
 swh/lister/cgit/lister.py              |  6 ++++++
 swh/lister/conda/lister.py             |  6 ++++++
 swh/lister/cpan/lister.py              |  6 ++++++
 swh/lister/cran/lister.py              | 11 ++++++++++-
 swh/lister/crates/lister.py            |  6 ++++++
 swh/lister/debian/lister.py            |  6 ++++++
 swh/lister/fedora/lister.py            |  8 +++++++-
 swh/lister/github/lister.py            |  6 ++++++
 swh/lister/gitlab/lister.py            | 17 ++++++++++++++++-
 swh/lister/gitlab/tests/test_lister.py | 35 ++++++++++++++++++++++++++++++++++
 swh/lister/gnu/lister.py               |  6 ++++++
 swh/lister/gogs/lister.py              |  6 ++++++
 swh/lister/golang/lister.py            |  6 ++++++
 swh/lister/hackage/lister.py           |  6 ++++++
 swh/lister/launchpad/lister.py         |  6 ++++++
 swh/lister/maven/lister.py             |  6 ++++++
 swh/lister/nixguix/lister.py           |  6 ++++++
 swh/lister/npm/lister.py               |  6 ++++++
 swh/lister/nuget/lister.py             |  6 ++++++
 swh/lister/opam/lister.py              |  6 ++++++
 swh/lister/packagist/lister.py         |  6 ++++++
 swh/lister/phabricator/lister.py       | 11 ++++++++++-
 swh/lister/pubdev/lister.py            |  6 ++++++
 swh/lister/puppet/lister.py            |  6 ++++++
 swh/lister/pypi/lister.py              |  6 ++++++
 swh/lister/rubygems/lister.py          |  6 ++++++
 swh/lister/sourceforge/lister.py       |  6 ++++++
 swh/lister/tuleap/lister.py            |  6 ++++++
 32 files changed, 240 insertions(+), 4 deletions(-)
Changes applied before test
commit e785e67315c63cdc4acca64e9c32e8e7e270074b
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Dec 5 16:33:45 2022 +0100

    Hook up recently introduced options to all listers
    
    Hopefully one day we'll be able to replace all of this mess with PEP692
    TypedDict kwargs, but that's only on track for Python 3.12.

commit 5ea79ee3e0deac0e83d2cb2ebb802181d6a5ed20
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Dec 5 15:36:40 2022 +0100

    gitlab: allow ignoring projects with certain path prefixes
    
    Some GitLab instances use specific namespaces for transient repositories
    that it doesn't make sense to archive (for example, gitlab.org has a set
    of QA namespaces used for integration testing of their production
    deployments; drupal has an `issues/` namespace with forks of repos that
    are only used for collaboration on merge requests, and aren't that
    useful to be archived).

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/874/ for more details.

olasd requested review of this revision.Dec 5 2022, 4:44 PM
This revision is now accepted and ready to land.Dec 5 2022, 5:05 PM