Differential D8607

nixguix: Deal with impossible communication with server
ClosedPublic
Actions

Authored by ardumont on Oct 4 2022, 2:08 PM.

Details

Reviewers

anlambert

Group Reviewers

Reviewers

Maniphest Tasks

T3781: Replace the Nixguix loader with a lister

Commits

rDLS06b11dd5f672: nixguix: Deal with impossible communication with server

Summary

When that arises, we skip the origins.

Related to T3781

Diff Detail

Repository

rDLS Listers

Branch

add-nixguix-lister

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 32062
Build 50200: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 50199: arc lint + arc unit

Event Timeline

ardumont created this revision.Oct 4 2022, 2:08 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptOct 4 2022, 2:08 PM

Build is green

Patch application report for D8607 (id=31092)

Could not rebase; Attempt merge onto fa1205c4df...

Updating fa1205c..06b11dd
Fast-forward
 requirements-swh.txt                               |   2 +-
 setup.py                                           |   1 +
 swh/lister/__init__.py                             |  22 ++
 swh/lister/gnu/tree.py                             |  21 +-
 swh/lister/nixguix/__init__.py                     |  38 ++
 swh/lister/nixguix/lister.py                       | 418 +++++++++++++++++++++
 swh/lister/nixguix/tasks.py                        |  14 +
 swh/lister/nixguix/tests/__init__.py               |   0
 .../nixguix/tests/data/guix-swh_sources.json       |  36 ++
 .../nixguix/tests/data/nixpkgs-swh_sources.json    |  64 ++++
 swh/lister/nixguix/tests/test_lister.py            | 270 +++++++++++++
 swh/lister/nixguix/tests/test_tasks.py             |  27 ++
 swh/lister/tests/test_cli.py                       |   4 +
 13 files changed, 899 insertions(+), 18 deletions(-)
 create mode 100644 swh/lister/nixguix/__init__.py
 create mode 100644 swh/lister/nixguix/lister.py
 create mode 100644 swh/lister/nixguix/tasks.py
 create mode 100644 swh/lister/nixguix/tests/__init__.py
 create mode 100644 swh/lister/nixguix/tests/data/guix-swh_sources.json
 create mode 100644 swh/lister/nixguix/tests/data/nixpkgs-swh_sources.json
 create mode 100644 swh/lister/nixguix/tests/test_lister.py
 create mode 100644 swh/lister/nixguix/tests/test_tasks.py

Changes applied before test

commit 06b11dd5f6728222c944da4881f4a84a739f33ac
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 14:07:42 2022 +0200

    nixguix: Deal with impossible communication with server
    
    When that arises, we skip the origins.
    
    Related to T3781

commit a94b75f366be5722c47022ce7afb55384bc8fbb6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 13:00:18 2022 +0200

    nixguix: Deal with mistyped origins
    
    Some origins are listed as urls while they are not. They are possibly vcs. So this
    commit tries to detect and and deal with those if possible. If not possible, they are
    skipped.
    
    Related to T3781
    Related to P1470

commit 1b4fe51f62c706a9ef77b8eea74e111bb8be3542
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 10:57:32 2022 +0200

    nixguix: Randomize order of listed origins
    
    The end goal is to ingest sparsely the origins, that would avoid hitting the various
    servers around the same time for colocated origins in the upstream manifest (especially
    file or tarball).
    
    Related to T3781

commit 94b6dbea0a7f602be0711a3bb1f9bb9e16fc48ce
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Oct 1 16:41:48 2022 +0200

    nixguix: Document lister
    
    Related to T3781

commit 6d2e7aa17808e39ba9f493b65d662d0ddef5796c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Oct 1 16:12:46 2022 +0200

    nixguix: Register task
    
    Related to T3781

commit fbfdf88ea4fe79c4846ecd48f2a1322f5d3995fc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Aug 30 11:17:33 2022 +0200

    nixguix: Add lister
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/741/ for more details.

Harbormaster completed remote builds in B32062: Diff 31092.Oct 4 2022, 2:14 PM

ardumont requested review of this revision.Oct 4 2022, 2:14 PM

ardumont mentioned this in T3781: Replace the Nixguix loader with a lister.Oct 4 2022, 2:19 PM

anlambert accepted this revision.Oct 4 2022, 2:33 PM

This revision is now accepted and ready to land.Oct 4 2022, 2:33 PM

Closed by commit rDLS06b11dd5f672: nixguix: Deal with impossible communication with server (authored by ardumont). · Explain WhyOct 4 2022, 2:37 PM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDLS06b11dd5f672: nixguix: Deal with impossible communication with server.