Page MenuHomeSoftware Heritage

nixguix: Deal with impossible communication with server
ClosedPublic

Authored by ardumont on Oct 4 2022, 2:08 PM.

Details

Summary

When that arises, we skip the origins.

Related to T3781

Diff Detail

Repository
rDLS Listers
Branch
add-nixguix-lister
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32062
Build 50200: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 50199: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D8607 (id=31092)

Could not rebase; Attempt merge onto fa1205c4df...

Updating fa1205c..06b11dd
Fast-forward
 requirements-swh.txt                               |   2 +-
 setup.py                                           |   1 +
 swh/lister/__init__.py                             |  22 ++
 swh/lister/gnu/tree.py                             |  21 +-
 swh/lister/nixguix/__init__.py                     |  38 ++
 swh/lister/nixguix/lister.py                       | 418 +++++++++++++++++++++
 swh/lister/nixguix/tasks.py                        |  14 +
 swh/lister/nixguix/tests/__init__.py               |   0
 .../nixguix/tests/data/guix-swh_sources.json       |  36 ++
 .../nixguix/tests/data/nixpkgs-swh_sources.json    |  64 ++++
 swh/lister/nixguix/tests/test_lister.py            | 270 +++++++++++++
 swh/lister/nixguix/tests/test_tasks.py             |  27 ++
 swh/lister/tests/test_cli.py                       |   4 +
 13 files changed, 899 insertions(+), 18 deletions(-)
 create mode 100644 swh/lister/nixguix/__init__.py
 create mode 100644 swh/lister/nixguix/lister.py
 create mode 100644 swh/lister/nixguix/tasks.py
 create mode 100644 swh/lister/nixguix/tests/__init__.py
 create mode 100644 swh/lister/nixguix/tests/data/guix-swh_sources.json
 create mode 100644 swh/lister/nixguix/tests/data/nixpkgs-swh_sources.json
 create mode 100644 swh/lister/nixguix/tests/test_lister.py
 create mode 100644 swh/lister/nixguix/tests/test_tasks.py
Changes applied before test
commit 06b11dd5f6728222c944da4881f4a84a739f33ac
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 14:07:42 2022 +0200

    nixguix: Deal with impossible communication with server
    
    When that arises, we skip the origins.
    
    Related to T3781

commit a94b75f366be5722c47022ce7afb55384bc8fbb6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 13:00:18 2022 +0200

    nixguix: Deal with mistyped origins
    
    Some origins are listed as urls while they are not. They are possibly vcs. So this
    commit tries to detect and and deal with those if possible. If not possible, they are
    skipped.
    
    Related to T3781
    Related to P1470

commit 1b4fe51f62c706a9ef77b8eea74e111bb8be3542
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 10:57:32 2022 +0200

    nixguix: Randomize order of listed origins
    
    The end goal is to ingest sparsely the origins, that would avoid hitting the various
    servers around the same time for colocated origins in the upstream manifest (especially
    file or tarball).
    
    Related to T3781

commit 94b6dbea0a7f602be0711a3bb1f9bb9e16fc48ce
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Oct 1 16:41:48 2022 +0200

    nixguix: Document lister
    
    Related to T3781

commit 6d2e7aa17808e39ba9f493b65d662d0ddef5796c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Oct 1 16:12:46 2022 +0200

    nixguix: Register task
    
    Related to T3781

commit fbfdf88ea4fe79c4846ecd48f2a1322f5d3995fc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Aug 30 11:17:33 2022 +0200

    nixguix: Add lister
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/741/ for more details.

This revision is now accepted and ready to land.Oct 4 2022, 2:33 PM