Page MenuHomeSoftware Heritage
Differential D8605

nixguix: Randomize order of listed origins
ClosedPublic
Actions

Authored by ardumont on Oct 4 2022, 10:59 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T3781: Replace the Nixguix loader with a lister
Commits
rDLS1b4fe51f62c7: nixguix: Randomize order of listed origins
Summary

The end goal is to ingest sparsely the origins, that would avoid hitting the various
servers around the same time for colocated origins in the upstream manifest (especially
file or tarball).

Related to T3781
Depends on D8341

Diff Detail

Repository
rDLS Listers
Branch
add-nixguix-lister
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32056
Build 50188: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 50187: arc lint + arc unit

Event Timeline

ardumont created this revision.Oct 4 2022, 10:59 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 4 2022, 10:59 AM
swh-public-ci added a comment.Oct 4 2022, 11:05 AM

Build is green

Patch application report for D8605 (id=31084)

Could not rebase; Attempt merge onto fa1205c4df...

Updating fa1205c..41eef0a
Fast-forward
 requirements-swh.txt                               |   2 +-
 setup.py                                           |   1 +
 swh/lister/__init__.py                             |  22 ++
 swh/lister/gnu/tree.py                             |  21 +-
 swh/lister/nixguix/__init__.py                     |  38 +++
 swh/lister/nixguix/lister.py                       | 374 +++++++++++++++++++++
 swh/lister/nixguix/tasks.py                        |  14 +
 swh/lister/nixguix/tests/__init__.py               |   0
 .../nixguix/tests/data/guix-swh_sources.json       |  19 ++
 .../nixguix/tests/data/nixpkgs-swh_sources.json    |  52 +++
 swh/lister/nixguix/tests/test_lister.py            | 244 ++++++++++++++
 swh/lister/nixguix/tests/test_tasks.py             |  27 ++
 swh/lister/tests/test_cli.py                       |   4 +
 13 files changed, 800 insertions(+), 18 deletions(-)
 create mode 100644 swh/lister/nixguix/__init__.py
 create mode 100644 swh/lister/nixguix/lister.py
 create mode 100644 swh/lister/nixguix/tasks.py
 create mode 100644 swh/lister/nixguix/tests/__init__.py
 create mode 100644 swh/lister/nixguix/tests/data/guix-swh_sources.json
 create mode 100644 swh/lister/nixguix/tests/data/nixpkgs-swh_sources.json
 create mode 100644 swh/lister/nixguix/tests/test_lister.py
 create mode 100644 swh/lister/nixguix/tests/test_tasks.py
Changes applied before test
commit 41eef0a80a64d896c8c556d39b7730c8d53b5669
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 10:57:32 2022 +0200

    nixguix: Randomize listed origins to ingest sparsely origins
    
    Especially the first time around, that would avoid hitting the various servers around
    the same time for grouped origins (especially file or tarball).
    
    Related to T3781

commit 94b6dbea0a7f602be0711a3bb1f9bb9e16fc48ce
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Oct 1 16:41:48 2022 +0200

    nixguix: Document lister
    
    Related to T3781

commit 6d2e7aa17808e39ba9f493b65d662d0ddef5796c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Oct 1 16:12:46 2022 +0200

    nixguix: Register task
    
    Related to T3781

commit fbfdf88ea4fe79c4846ecd48f2a1322f5d3995fc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Aug 30 11:17:33 2022 +0200

    nixguix: Add lister
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/734/ for more details.

Harbormaster completed remote builds in B32054: Diff 31084.Oct 4 2022, 11:05 AM
ardumont requested review of this revision.Oct 4 2022, 11:05 AM
ardumont added inline comments.Oct 4 2022, 11:06 AM
swh/lister/nixguix/lister.py
376

I did not push it in lister.pattern directly but i wondered...

vlorentz added a subscriber: vlorentz.Oct 4 2022, 11:08 AM

What is random about this?

ardumont added a comment.Oct 4 2022, 11:17 AM

lol, nothing...

I forgot the main gist of it

batch_origins.shuffle()
yield batch_origins
vlorentz added a comment.Oct 4 2022, 11:17 AM

and you don't need grouper() for that

vlorentz added a comment.Oct 4 2022, 11:18 AM

by the way, you should reword the commit/diff title to mention the order is randomized, not origins themselves

ardumont retitled this revision from nixguix: Randomize listed origins to ingest sparsely origins to nixguix: Randomize order of listed origins.Oct 4 2022, 11:30 AM
ardumont edited the summary of this revision. (Show Details)
ardumont updated this revision to Diff 31086.Oct 4 2022, 11:39 AM

Actually randomize and do it at the manifest reading step.

ardumont added a comment.Oct 4 2022, 11:40 AM

and you don't need grouper() for that

Totally. Amended.

by the way, you should reword the commit/diff title to mention the order is randomized, not origins themselves

Indeed. Fixed.

Thanks.

swh-public-ci added a comment.Oct 4 2022, 11:46 AM

Build is green

Patch application report for D8605 (id=31086)

Could not rebase; Attempt merge onto fa1205c4df...

Updating fa1205c..cdeb7f5
Fast-forward
 requirements-swh.txt                               |   2 +-
 setup.py                                           |   1 +
 swh/lister/__init__.py                             |  22 ++
 swh/lister/gnu/tree.py                             |  21 +-
 swh/lister/nixguix/__init__.py                     |  38 +++
 swh/lister/nixguix/lister.py                       | 374 +++++++++++++++++++++
 swh/lister/nixguix/tasks.py                        |  14 +
 swh/lister/nixguix/tests/__init__.py               |   0
 .../nixguix/tests/data/guix-swh_sources.json       |  19 ++
 .../nixguix/tests/data/nixpkgs-swh_sources.json    |  52 +++
 swh/lister/nixguix/tests/test_lister.py            | 244 ++++++++++++++
 swh/lister/nixguix/tests/test_tasks.py             |  27 ++
 swh/lister/tests/test_cli.py                       |   4 +
 13 files changed, 800 insertions(+), 18 deletions(-)
 create mode 100644 swh/lister/nixguix/__init__.py
 create mode 100644 swh/lister/nixguix/lister.py
 create mode 100644 swh/lister/nixguix/tasks.py
 create mode 100644 swh/lister/nixguix/tests/__init__.py
 create mode 100644 swh/lister/nixguix/tests/data/guix-swh_sources.json
 create mode 100644 swh/lister/nixguix/tests/data/nixpkgs-swh_sources.json
 create mode 100644 swh/lister/nixguix/tests/test_lister.py
 create mode 100644 swh/lister/nixguix/tests/test_tasks.py
Changes applied before test
commit cdeb7f59ed72ebda661cab55a14556ece8905d64
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 10:57:32 2022 +0200

    nixguix: Randomize order of listed origins
    
    The end goal is to ingest sparsely the origins, that would avoid hitting the various
    servers around the same time for colocated origins in the upstream manifest (especially
    file or tarball).
    
    Related to T3781

commit 94b6dbea0a7f602be0711a3bb1f9bb9e16fc48ce
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Oct 1 16:41:48 2022 +0200

    nixguix: Document lister
    
    Related to T3781

commit 6d2e7aa17808e39ba9f493b65d662d0ddef5796c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Oct 1 16:12:46 2022 +0200

    nixguix: Register task
    
    Related to T3781

commit fbfdf88ea4fe79c4846ecd48f2a1322f5d3995fc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Aug 30 11:17:33 2022 +0200

    nixguix: Add lister
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/735/ for more details.

Harbormaster completed remote builds in B32056: Diff 31086.Oct 4 2022, 11:46 AM
vlorentz accepted this revision.Oct 4 2022, 11:47 AM
vlorentz added inline comments.
swh/lister/nixguix/lister.py
269

More readable, IMO.

(and remove the import)

This revision is now accepted and ready to land.Oct 4 2022, 11:47 AM
ardumont updated this revision to Diff 31087.Oct 4 2022, 11:54 AM

Adapt according to review.
That makes sense plus i forgot i already had that import so fine!

Thanks again

ardumont marked an inline comment as done.Oct 4 2022, 11:54 AM
swh-public-ci added a comment.Oct 4 2022, 12:00 PM

Build is green

Patch application report for D8605 (id=31087)

Could not rebase; Attempt merge onto fa1205c4df...

Updating fa1205c..1b4fe51
Fast-forward
 requirements-swh.txt                               |   2 +-
 setup.py                                           |   1 +
 swh/lister/__init__.py                             |  22 ++
 swh/lister/gnu/tree.py                             |  21 +-
 swh/lister/nixguix/__init__.py                     |  38 +++
 swh/lister/nixguix/lister.py                       | 373 +++++++++++++++++++++
 swh/lister/nixguix/tasks.py                        |  14 +
 swh/lister/nixguix/tests/__init__.py               |   0
 .../nixguix/tests/data/guix-swh_sources.json       |  19 ++
 .../nixguix/tests/data/nixpkgs-swh_sources.json    |  52 +++
 swh/lister/nixguix/tests/test_lister.py            | 244 ++++++++++++++
 swh/lister/nixguix/tests/test_tasks.py             |  27 ++
 swh/lister/tests/test_cli.py                       |   4 +
 13 files changed, 799 insertions(+), 18 deletions(-)
 create mode 100644 swh/lister/nixguix/__init__.py
 create mode 100644 swh/lister/nixguix/lister.py
 create mode 100644 swh/lister/nixguix/tasks.py
 create mode 100644 swh/lister/nixguix/tests/__init__.py
 create mode 100644 swh/lister/nixguix/tests/data/guix-swh_sources.json
 create mode 100644 swh/lister/nixguix/tests/data/nixpkgs-swh_sources.json
 create mode 100644 swh/lister/nixguix/tests/test_lister.py
 create mode 100644 swh/lister/nixguix/tests/test_tasks.py
Changes applied before test
commit 1b4fe51f62c706a9ef77b8eea74e111bb8be3542
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 10:57:32 2022 +0200

    nixguix: Randomize order of listed origins
    
    The end goal is to ingest sparsely the origins, that would avoid hitting the various
    servers around the same time for colocated origins in the upstream manifest (especially
    file or tarball).
    
    Related to T3781

commit 94b6dbea0a7f602be0711a3bb1f9bb9e16fc48ce
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Oct 1 16:41:48 2022 +0200

    nixguix: Document lister
    
    Related to T3781

commit 6d2e7aa17808e39ba9f493b65d662d0ddef5796c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Oct 1 16:12:46 2022 +0200

    nixguix: Register task
    
    Related to T3781

commit fbfdf88ea4fe79c4846ecd48f2a1322f5d3995fc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Aug 30 11:17:33 2022 +0200

    nixguix: Add lister
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/736/ for more details.

Harbormaster completed remote builds in B32057: Diff 31087.Oct 4 2022, 12:00 PM
ardumont added a child revision: D8606: nixguix: Deal with mistyped origins.Oct 4 2022, 1:02 PM
ardumont mentioned this in T3781: Replace the Nixguix loader with a lister.Oct 4 2022, 2:19 PM
Closed by commit rDLS1b4fe51f62c7: nixguix: Randomize order of listed origins (authored by ardumont). · Explain WhyOct 4 2022, 2:37 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDLS1b4fe51f62c7: nixguix: Randomize order of listed origins.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
lister/
nixguix/
6 lines100%
Loading...

Diff 31086

View Options

swh/lister/nixguix/lister.py

Loading...
About · Developer information · Legal