cli: Add more useful defaults
Changes PlannedPublicDraft
Actions

Authored by vlorentz on Jan 6 2023, 3:38 PM.

Details

Reviewers

None

Group Reviewers

Summary

two paths + reimport tasks at the package level so they are automatically
picked up by luigi without passing all module names on the CLI

Depends on D9006.

Diff Detail

Repository

rDGRPH Compressed graph representation

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 33441
Build 52422: Phabricator diff pipeline on jenkins
Build 52421: arc lint + arc unit

Event Timeline

vlorentz created this revision.Jan 6 2023, 3:38 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJan 6 2023, 3:38 PM

vlorentz added a child revision: D9008: luigi: Add an option to define the maximum RAM used by graph compression.Jan 6 2023, 3:38 PM

Build was aborted

Patch application report for D9007 (id=32473)

Could not rebase; Attempt merge onto f87e0a3c3c...

Updating f87e0a3..457c5c0
Fast-forward
 .../graph/rpc/NodePropertyBuilder.java             | 15 +++-
 .../java/org/softwareheritage/graph/GraphTest.java |  1 +
 .../org/softwareheritage/graph/SubgraphTest.java   | 13 ++--
 .../softwareheritage/graph/rpc/CountEdgesTest.java |  6 +-
 .../softwareheritage/graph/rpc/CountNodesTest.java |  8 +--
 .../graph/rpc/FindPathBetweenTest.java             |  2 +-
 .../softwareheritage/graph/rpc/FindPathToTest.java | 21 ++++--
 .../softwareheritage/graph/rpc/GetNodeTest.java    | 82 +++++++++++++---------
 .../org/softwareheritage/graph/rpc/StatsTest.java  |  6 +-
 .../graph/rpc/TraverseLeavesTest.java              |  2 +
 .../graph/rpc/TraverseNeighborsTest.java           |  2 +
 .../graph/rpc/TraverseNodesTest.java               | 20 +++---
 swh/graph/cli.py                                   | 32 ++++++++-
 swh/graph/luigi/__init__.py                        |  8 ++-
 tox.ini                                            |  6 +-
 15 files changed, 147 insertions(+), 77 deletions(-)

Changes applied before test

commit 457c5c0ebda8fcfb3c8e8396b29e0db455982025
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 6 11:04:05 2023 +0100

    cli: Add more useful defaults
    
    two paths + reimport tasks at the package level so they are automatically
    picked up by luigi without passing all module names on the CLI

commit 5b8751646d769a28ca8f7b105a102948605a1ea7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 6 10:55:26 2023 +0100

    cli: Add flag --s3-athena-output-location to configure all Luigi tasks at once

commit 03c8ca6a6272d7f7cd6cd3b75ab5bfcaf4538fe6
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 5 18:37:43 2023 +0100

    Fix Java tests broken by 559d4068bfe1dd50d57062192c0e22664ada03c8

commit d344071c0f6ee1d38a368a34f476fdbf15257d91
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 6 15:29:37 2023 +0100

    tox: s/whitelist_externals/allowlist_externals/
    
    whitelist_externals is deprecated since v3.18, and removed since v4.0

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/353/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/353/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 6 2023, 3:51 PM

Harbormaster failed remote builds in B33441: Diff 32473!