Page MenuHomeSoftware Heritage

Use AllowedNodesTest to implement return type filtering
ClosedPublic

Authored by seirl on Jan 15 2022, 12:03 AM.

Details

Summary

Old implementation used a new NodesFiltering class for no apparent
reason, whereas something was already implemented to parse node filter
strings.
Plus, the old implementation would only filter the results of the eager
functions, instead of filtering upstream before the visitor callbacks
were called, which made it not work for most use cases.

Fix T2981

Diff Detail

Repository
rDGRPH Compressed graph representation
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D6954 (id=25203)

Could not rebase; Attempt merge onto 64f35108dd...

Updating 64f3510..646c05d
Fast-forward
 java/pom.xml                                       |  79 +++++-
 .../org/softwareheritage/graph/AllowedNodes.java   |   2 +-
 .../graph/BidirectionalImmutableGraph.java         |   2 +-
 .../java/org/softwareheritage/graph/Entry.java     |  10 +-
 .../java/org/softwareheritage/graph/Graph.java     | 304 ---------------------
 .../org/softwareheritage/graph/NodesFiltering.java | 107 --------
 .../java/org/softwareheritage/graph/Subgraph.java  |   4 +-
 .../graph/SwhBidirectionalGraph.java               | 166 +++++++++++
 .../java/org/softwareheritage/graph/SwhGraph.java  |  47 ++++
 .../softwareheritage/graph/SwhGraphProperties.java |  85 ++++++
 .../graph/SwhUnidirectionalGraph.java              | 221 +++++++++++++++
 .../java/org/softwareheritage/graph/Traversal.java | 114 +++++---
 .../graph/algo/TopologicalTraversal.java           |   4 +-
 .../graph/benchmark/AccessEdge.java                |   4 +-
 .../org/softwareheritage/graph/benchmark/BFS.java  |   4 +-
 .../graph/benchmark/Benchmark.java                 |   6 +-
 .../softwareheritage/graph/benchmark/Browsing.java |   4 +-
 .../graph/benchmark/Provenance.java                |   4 +-
 .../softwareheritage/graph/benchmark/Vault.java    |   4 +-
 .../graph/benchmark/utils/Random.java              |   6 +-
 .../experiments/forks/FindCommonAncestor.java      |   6 +-
 .../graph/experiments/forks/FindPath.java          |   6 +-
 .../graph/experiments/forks/ForkCC.java            |   6 +-
 .../graph/experiments/forks/ForkCliques.java       |   6 +-
 .../graph/experiments/forks/ListEmptyOrigins.java  |   6 +-
 .../multiplicationfactor/GenDistribution.java      |   8 +-
 .../graph/experiments/topology/AveragePaths.java   |   6 +-
 .../topology/ClusteringCoefficient.java            |  12 +-
 .../experiments/topology/ConnectedComponents.java  |   4 +-
 .../graph/experiments/topology/InOutDegree.java    |  10 +-
 .../topology/SubdatasetSizeFunction.java           |   6 +-
 .../graph/maps/LabelMapBuilder.java                |   2 +-
 .../org/softwareheritage/graph/maps/MapFile.java   |   4 +
 .../org/softwareheritage/graph/maps/NodeIdMap.java |  10 +-
 .../org/softwareheritage/graph/server/App.java     |   6 +-
 .../softwareheritage/graph/server/Endpoint.java    |   4 +-
 .../graph/utils/ExportSubdataset.java              |   4 +-
 .../graph/utils/FindEarliestRevision.java          |   9 +-
 .../softwareheritage/graph/utils/ReadGraph.java    |   2 +-
 .../graph/utils/ReadLabelledGraph.java             |   2 +-
 .../softwareheritage/graph/AllowedNodesTest.java   |  53 ++++
 .../java/org/softwareheritage/graph/GraphTest.java |   6 +-
 .../org/softwareheritage/graph/LeavesTest.java     |  12 +-
 .../org/softwareheritage/graph/NeighborsTest.java  |   8 +-
 .../org/softwareheritage/graph/SubgraphTest.java   |  10 +-
 .../java/org/softwareheritage/graph/VisitTest.java |  28 +-
 .../java/org/softwareheritage/graph/WalkTest.java  |  14 +-
 47 files changed, 843 insertions(+), 584 deletions(-)
 delete mode 100644 java/src/main/java/org/softwareheritage/graph/Graph.java
 delete mode 100644 java/src/main/java/org/softwareheritage/graph/NodesFiltering.java
 create mode 100644 java/src/main/java/org/softwareheritage/graph/SwhBidirectionalGraph.java
 create mode 100644 java/src/main/java/org/softwareheritage/graph/SwhGraph.java
 create mode 100644 java/src/main/java/org/softwareheritage/graph/SwhGraphProperties.java
 create mode 100644 java/src/main/java/org/softwareheritage/graph/SwhUnidirectionalGraph.java
 create mode 100644 java/src/test/java/org/softwareheritage/graph/AllowedNodesTest.java
Changes applied before test
commit 646c05dfd5c4e8544837c41ac7f22047e56c4a76
Author: Antoine Pietri <antoine.pietri1@gmail.com>
Date:   Sat Jan 15 00:00:45 2022 +0100

    Use AllowedNodesTest to implement return type filtering
    
    Old implementation used a new NodesFiltering class for no apparent
    reason, whereas something was already implemented to parse node filter
    strings.
    Plus, the old implementation would only filter the results of the eager
    functions, instead of filtering upstream before the visitor callbacks
    were called, which made it not work for most use cases.
    
    Fix T2981

commit 83fcf6bb815618ef711383d4bd17a4436caca0aa
Author: Antoine Pietri <antoine.pietri1@gmail.com>
Date:   Sat Dec 4 00:50:57 2021 +0100

    Refactor Graph class in SwhUnidirectionalGraph and SwhBidirectionalGraph
    
    This commit is a massive refactor of the project to apply a stronger
    separation of concerns which will allow the project to be more
    maintainable and extensible in the future.
    
    The principal change is the switch from a single SwhGraph containing
    everything SWH-specific to two new classes:
    
    - SwhUnidirectionalGraph, an ImmutableGraph augmented with SWH-specific
      methods such as getSWHID(), getNodeType(), etc.
    
    - SwhBidirectionalGraph, which leverages BidirectionalImmutableGraph to
      store two SwhUnidirectionnalGraph, one for each direction.
    
    To share common behavior and storage between these two classes, such as
    handling node types and node properties, another class has been created:
    
    - SwhGraphProperties, which is designed to contain the graph properties
      that can apply on the graph whatever its direction is (node types,
      node properties, dataset version, etc.)
    
    Finally, to allow users of this API to use the two classes
    interchangeably when the direction(s) do not mattern, both classes
    implement a new interface:
    
    - SwhGraph, which is designed to hold the common SWH-specific
      behavior between SwhUnidirectionalGraph and SwhBidirectionalGraph.
    
                        ┌──────────────┐
                        │ImmutableGraph◄────────┐
                        └────▲─────────┘        │extends
                             │                  │
                             │       ┌──────────┴────────────────┐
                      extends│       │BidirectionalImmutableGraph│
                             │       └────────────▲──────────────┘
                             │                    │extends
              ┌──────────────┴───────┐     ┌──────┴──────────────┐
              │SwhUnidirectionalGraph│◄────┤SwhBidirectionalGraph│
              └──┬──────────────┬────┘     └────────┬───────────┬┘
                 │              │    contains x2    │           │
                 │              │                   │           │
                 │    implements│                   │implements │
                 │             ┌▼──────────┐        │           │
                 │             │SwhGraph(I)◄────────┘           │
        contains │             └───────────┘                    │contains
                 │                                              │
                 │            ┌──────────────────┐              │
                 └────────────►SwhGraphProperties◄──────────────┘
                              └──────────────────┘
    
    BidirectionalImmutableGraph is included in this pull-request, but is
    intended to be merged upstream eventually:
    
    - https://github.com/vigna/webgraph/pull/5
    - https://github.com/vigna/webgraph-big/pull/2
    
    This commit includes a (partial) documentation overhaul. Notably,
    using cross-dependency compiling of java documentation allows us to
    remove redundant method documentations when they are inherited from
    webgraph classes. It also includes cross-dependency *linking*, which
    allows users to click on external links to unimi.it libraries.
    
    Finally, this commits introduces stub methods for optionally loading
    *labelled* graphs, which will be useful to store edges properties such
    as file names, etc. To avoid a combinatorial explosion of types
    (SwhUnidirectionalLabelledGraph, SwhBidirectionalLabelledGraph, etc..),
    this is only checked at runtime. Classes hold both the unlabelled and
    labelled versions of the underlying graphs, with the labelled version
    being equal to null when the labels have not been loaded.

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/156/ for more details.

Build is green

Patch application report for D6954 (id=25212)

Rebasing onto bb9b5fc59d...

Current branch diff-target is up to date.
Changes applied before test
commit 294128e0f96e12ab0dd19cb5d98452729f0119b9
Author: Antoine Pietri <antoine.pietri1@gmail.com>
Date:   Sat Jan 15 00:00:45 2022 +0100

    Use AllowedNodesTest to implement return type filtering
    
    Old implementation used a new NodesFiltering class for no apparent
    reason, whereas something was already implemented to parse node filter
    strings.
    Plus, the old implementation would only filter the results of the eager
    functions, instead of filtering upstream before the visitor callbacks
    were called, which made it not work for most use cases.
    
    Fix T2981

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/158/ for more details.

This revision is now accepted and ready to land.Jan 18 2022, 10:24 AM