Differential D9008

luigi: Add an option to define the maximum RAM used by graph compression
Changes PlannedPublicDraft
Actions

Authored by vlorentz on Jan 6 2023, 3:38 PM.

Details

Reviewers

None

Group Reviewers

Reviewers

Summary

Depends on D9007.

Diff Detail

Repository

rDGRPH Compressed graph representation

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 33442
Build 52424: Phabricator diff pipeline on jenkins
Build 52423: arc lint + arc unit

Event Timeline

vlorentz created this revision.Jan 6 2023, 3:38 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJan 6 2023, 3:38 PM

vlorentz added a child revision: D9009: compression: Force log level to be either DEBUG or INFO.Jan 6 2023, 3:38 PM

Build was aborted

Patch application report for D9008 (id=32474)

Could not rebase; Attempt merge onto f87e0a3c3c...

Updating f87e0a3..2d44367
Fast-forward
 .../graph/rpc/NodePropertyBuilder.java             | 15 +++-
 .../java/org/softwareheritage/graph/GraphTest.java |  1 +
 .../org/softwareheritage/graph/SubgraphTest.java   | 13 ++--
 .../softwareheritage/graph/rpc/CountEdgesTest.java |  6 +-
 .../softwareheritage/graph/rpc/CountNodesTest.java |  8 +--
 .../graph/rpc/FindPathBetweenTest.java             |  2 +-
 .../softwareheritage/graph/rpc/FindPathToTest.java | 21 ++++--
 .../softwareheritage/graph/rpc/GetNodeTest.java    | 82 +++++++++++++---------
 .../org/softwareheritage/graph/rpc/StatsTest.java  |  6 +-
 .../graph/rpc/TraverseLeavesTest.java              |  2 +
 .../graph/rpc/TraverseNeighborsTest.java           |  2 +
 .../graph/rpc/TraverseNodesTest.java               | 20 +++---
 swh/graph/cli.py                                   | 32 ++++++++-
 swh/graph/luigi/__init__.py                        |  8 ++-
 swh/graph/luigi/compressed_graph.py                |  3 +
 tox.ini                                            |  6 +-
 16 files changed, 150 insertions(+), 77 deletions(-)

Changes applied before test

commit 2d44367afd2742db4f74e6f498c9da1c4e9ce892
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 6 11:05:11 2023 +0100

    luigi: Add an option to define the maximum RAM used by graph compression

commit 457c5c0ebda8fcfb3c8e8396b29e0db455982025
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 6 11:04:05 2023 +0100

    cli: Add more useful defaults
    
    two paths + reimport tasks at the package level so they are automatically
    picked up by luigi without passing all module names on the CLI

commit 5b8751646d769a28ca8f7b105a102948605a1ea7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 6 10:55:26 2023 +0100

    cli: Add flag --s3-athena-output-location to configure all Luigi tasks at once

commit 03c8ca6a6272d7f7cd6cd3b75ab5bfcaf4538fe6
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 5 18:37:43 2023 +0100

    Fix Java tests broken by 559d4068bfe1dd50d57062192c0e22664ada03c8

commit d344071c0f6ee1d38a368a34f476fdbf15257d91
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 6 15:29:37 2023 +0100

    tox: s/whitelist_externals/allowlist_externals/
    
    whitelist_externals is deprecated since v3.18, and removed since v4.0

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/354/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/354/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 6 2023, 3:52 PM

Harbormaster failed remote builds in B33442: Diff 32474!