Page Menu
Home
Software Heritage
Search
Configure Global Search
Log In
Files
F7066369
D7890.id28932.diff
No One
Temporary
Actions
View File
Edit File
Delete File
View Transforms
Subscribe
Mute Notifications
Award Token
Flag For Later
Size
405 KB
Subscribers
None
D7890.id28932.diff
View Options
This file is larger than 256 KB, so syntax highlighting was skipped.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -17,7 +17,7 @@
hooks:
- id: codespell
name: Check source code spelling
- args: ["-L te,wth,alledges"]
+ args: ["-L te,wth,alledges,afterall"]
stages: [commit]
- repo: local
@@ -48,3 +48,5 @@
args: ["-f", "java/pom.xml", "spotless:apply"]
pass_filenames: false
language: system
+
+exclude: ^swh/graph/rpc/
diff --git a/Makefile.local b/Makefile.local
--- a/Makefile.local
+++ b/Makefile.local
@@ -9,6 +9,9 @@
java-%:
mvn -f $(POM_PATH) $*
+protoc:
+ python -m grpc_tools.protoc -I. --python_out=. --mypy_out=. --grpc_python_out=. swh/graph/rpc/*.proto
+
clean-java: java-clean
.PHONY: java clean-java
diff --git a/docs/grpc-api.rst b/docs/grpc-api.rst
new file mode 100644
--- /dev/null
+++ b/docs/grpc-api.rst
@@ -0,0 +1,553 @@
+.. _swh-graph-grpc-api:
+
+==================
+Using the GRPC API
+==================
+
+The GRPC API is the core API used to query the graph remotely. It uses the
+`GRPC framework <https://grpc.io/>`_ to provide high-performance graph
+traversal methods with server streaming.
+
+It is more expressive than the :ref:`HTTP API <swh-graph-api>` (which itself
+uses the GRPC API under the hood to serve queries), however it can only be
+used internally or with a local setup, and is never exposed publicly.
+
+Its major features include: returning node and edge properties, performing BFS
+traversals, including traversals with more than one starting node, finding
+shortest paths, common ancestors, etc.
+
+Quickstart
+==========
+
+Starting the server
+-------------------
+
+The GRPC server is automatically started on port 50091 when the HTTP server
+is started with ``swh graph rpc-serve``. It can also be started directly with
+Java, instead of going through the Python layer, by using the fat-jar shipped
+with swh-graph:
+
+.. code-block:: console
+
+ $ java -cp swh-graph-XXX.jar org.softwareheritage.graph.rpc.GraphServer <graph_basename>
+
+(See :ref:`swh-graph-java-api` and :ref:`swh-graph-memory` for more
+information on Java process options and JVM tuning.)
+
+Running queries
+---------------
+
+The `gRPC command line tool
+<https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md>`_
+can be an easy way to query the GRPC API from the command line. It is
+invoked with the ``grpc_cli`` command. Of course, it is also possible to use
+a generated RPC client in any programming language supported by GRPC.
+
+All RPC methods are defined in the service ``swh.graph.TraversalService``.
+The available endpoints can be listed with ``ls``:
+
+.. code-block:: console
+
+ $ grpc_cli ls localhost:50091 swh.graph.TraversalService
+ Traverse
+ FindPathTo
+ FindPathBetween
+ CountNodes
+ CountEdges
+ Stats
+ GetNode
+
+A RPC method can be called with the ``call`` subcommand.
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.Stats ""
+ connecting to localhost:50091
+ num_nodes: 21
+ num_edges: 23
+ compression: 1.412
+ bits_per_node: 8.524
+ [...]
+ Rpc succeeded with OK status
+
+The ``--json-output`` flag can also be used to make the results easier to
+parse.
+
+.. code-block:: console
+
+ $ grpc_cli --json_output call localhost:50091 swh.graph.TraversalService.Stats ""
+ connecting to localhost:50091
+ {
+ "numNodes": "21",
+ "numEdges": "23",
+ [...]
+ }
+ Rpc succeeded with OK status
+
+
+**Note**: grpc_cli's outputs in this document are slightly modified for
+readability's sake.
+
+Simple queries
+==============
+
+For a full documentation of all the endpoints, as well as the request and
+response messages, see :ref:`swh-graph-grpc-api-protobuf`.
+
+Querying a single node
+----------------------
+
+The **GetNode** endpoint can be used to return information on a single
+node of the graph, including all its node properties, from its SWHID. Here
+are a few examples from the test graph:
+
+Content
+~~~~~~~
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.GetNode \
+ 'swhid: "swh:1:cnt:0000000000000000000000000000000000000001"'
+
+.. code-block:: javascript
+
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000001"
+ cnt {
+ length: 42
+ is_skipped: false
+ }
+
+Revision
+~~~~~~~~
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.GetNode \
+ 'swhid: "swh:1:rev:0000000000000000000000000000000000000009"'
+
+.. code-block:: javascript
+
+ swhid: "swh:1:rev:0000000000000000000000000000000000000009"
+ rev {
+ author: 2
+ author_date: 1111140840
+ author_date_offset: 120
+ committer: 2
+ committer_date: 1111151950
+ committer_date_offset: 120
+ message: "Add parser"
+ }
+
+Release
+~~~~~~~
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.GetNode \
+ 'swhid: "swh:1:rel:0000000000000000000000000000000000000010"'
+
+.. code-block:: javascript
+
+ swhid: "swh:1:rel:0000000000000000000000000000000000000010"
+ rel {
+ author: 0
+ author_date: 1234564290
+ author_date_offset: 120
+ message: "Version 1.0"
+ }
+
+Origin
+~~~~~~
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.GetNode \
+ 'swhid: "swh:1:ori:83404f995118bd25774f4ac14422a8f175e7a054"'
+
+.. code-block:: javascript
+
+ swhid: "swh:1:ori:83404f995118bd25774f4ac14422a8f175e7a054"
+ ori {
+ url: "https://example.com/swh/graph"
+ }
+
+
+Checking the presence of a node
+-------------------------------
+
+The **GetNode** endpoint can also be used to check if a node exists in the
+graph. The RPC will return the ``INVALID_ARGUMENT`` code, and a detailed error
+message.
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.GetNode \
+ 'swhid: "swh:1:ori:ffffffffffffffffffffffffffffffffffffffff"'
+ Rpc failed with status code 3, error message: Unknown SWHID: swh:1:ori:ffffffffffffffffffffffffffffffffffffffff
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.GetNode \
+ 'swhid: "invalidswhid"'
+ Rpc failed with status code 3, error message: malformed SWHID: swh:1:ori:ffffffffffffffffffffffffffffffffffffffff
+
+
+Selecting returned fields with FieldMask
+----------------------------------------
+
+Many endpoints, including **GetNode**, contain a ``mask`` field of type
+`FieldMask
+<https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/FieldMask.html>`_,
+which can be used to select which fields should be returned in the response.
+
+This is particularly interesting for traversal queries that return a large
+number of nodes, because property access is quite costly from the compressed
+graph (at least compared to regular node access). It is therefore recommended
+that clients systematically use FieldMasks to only request the properties that
+they will consume.
+
+A FieldMask is represented as a set of "field paths" in dotted notation. For
+instance, ``paths: ["swhid", "rev.message"]`` will only request the swhid and
+the message of a given node. An empty mask will return an empty object.
+
+Example:
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.GetNode \
+ 'swhid: "swh:1:rev:0000000000000000000000000000000000000009", mask: {paths: ["swhid"]}'
+ swhid: "swh:1:rev:0000000000000000000000000000000000000009"
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.GetNode \
+ 'swhid: "swh:1:rev:0000000000000000000000000000000000000009", mask: {paths: ["swhid", "rev.message", "rev.author"]}'
+ swhid: "swh:1:rev:0000000000000000000000000000000000000009"
+ rev {
+ author: 2
+ message: "Add parser"
+ }
+
+
+Getting statistics on the graph
+-------------------------------
+
+The **Stats** endpoint returns overall statistics on the entire compressed
+graph. Most notably, the total number of nodes and edges, as well as the
+range of indegrees and outdegrees, and some compression-related statistics.
+
+.. code-block:: console
+
+ $ grpc_cli --json_output call localhost:50091 swh.graph.TraversalService.Stats ""
+
+.. code-block:: json
+
+ {
+ "numNodes": "21",
+ "numEdges": "23",
+ "compression": 1.412,
+ "bitsPerNode": 8.524,
+ "bitsPerEdge": 7.783,
+ "avgLocality": 2.522,
+ "indegreeMax": "3",
+ "indegreeAvg": 1.0952380952380953,
+ "outdegreeMax": "3",
+ "outdegreeAvg": 1.0952380952380953
+ }
+
+
+Graph traversals
+================
+
+Breadth-first traversal
+-----------------------
+
+The **Traverse** endpoint performs a breadth-first traversal from a set of
+source nodes, and `streams
+<https://grpc.io/docs/what-is-grpc/core-concepts/#server-streaming-rpc>`_ all
+the nodes it encounters on the way. All the node properties are stored in the
+result nodes. Additionally, the *edge properties* (e.g., directory entry names
+and permissions) are stored as a list in the ``successor`` field of each node.
+
+For instance, here we run a traversal from a directory that contains two
+contents:
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.Traverse \
+ "src: 'swh:1:dir:0000000000000000000000000000000000000006'"
+
+We get the following stream of nodes: first, the source directory (including
+its properties, successor list and their labels), then the contents themselves
+and their respective properties.
+
+.. code-block:: javascript
+
+ swhid: "swh:1:dir:0000000000000000000000000000000000000006"
+ successor {
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000005"
+ label {
+ name: "parser.c"
+ permission: 33188
+ }
+ }
+ successor {
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000004"
+ label {
+ name: "README.md"
+ permission: 33188
+ }
+ }
+ num_successors: 2
+
+.. code-block:: javascript
+
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000005"
+ cnt {
+ length: 1337
+ is_skipped: false
+ }
+
+.. code-block:: javascript
+
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000004"
+ cnt {
+ length: 404
+ is_skipped: false
+ }
+
+Again, it is possible to use a FieldMask to restrict which fields get returned.
+For instance, if we only care about the SWHIDs:
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.Traverse \
+ "src: 'swh:1:dir:0000000000000000000000000000000000000006', mask: {paths: ['swhid']}"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000006"
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000005"
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000004"
+
+
+Graph direction
+~~~~~~~~~~~~~~~
+
+For many purposes, especially that of finding the provenance of software
+artifacts, it is useful to query the backward (or transposed) graph instead,
+which is the same as the forward graph except all the edges are reversed.
+To achieve this, the ``direction`` field can be used to specify a direction
+from the ``GraphDirection`` enum (either ``FORWARD`` or ``BACKWARD``).
+
+This query returns all the nodes reachable from a given directory in the
+*backward* (or "transposed") graph:
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.Traverse \
+ "src: 'swh:1:dir:0000000000000000000000000000000000000006', direction: BACKWARD, mask: {paths: ['swhid']}"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000006"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000008"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000012"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000009"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000013"
+ swhid: "swh:1:rel:0000000000000000000000000000000000000010"
+ swhid: "swh:1:snp:0000000000000000000000000000000000000020"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000018"
+ swhid: "swh:1:ori:83404f995118bd25774f4ac14422a8f175e7a054"
+ swhid: "swh:1:rel:0000000000000000000000000000000000000019"
+
+
+Edge restrictions
+~~~~~~~~~~~~~~~~~
+
+To constrain the types of edges that can be followed during the graph
+traversal, it is possible to specify an edge restriction string in the ``edge``
+field. It is a comma-separated list of edge types that will be followed (e.g.
+``"rev:dir,dir:cnt"`` to only follow revision → directory and directory →
+content edges).
+By default (or when ``"*"`` is provided), all edges can be followed.
+
+This query traverses the parent revisions of a given revision only (i.e., it
+outputs the *commit log* from a given commit):
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.Traverse \
+ "src: 'swh:1:rev:0000000000000000000000000000000000000018', edges: 'rev:rev', mask: {paths: ['swhid']}"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000018"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000013"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000009"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000003"
+
+
+Limiting the traversal
+~~~~~~~~~~~~~~~~~~~~~~
+
+To avoid using up too much memory or resources, a traversal can be limited
+in two different ways:
+
+- the ``max_depth`` attribute defines the maximum depth of the traversal.
+- the ``max_edges`` attribute defines the maximum number of edges that can be
+ fetched by the traversal.
+
+When these limits are reached, the traversal will simply stop. While these
+options have obvious use-cases for anti-abuse, they can also be semantically
+useful: for instance, specifying ``max_depth: 1`` will only return the
+*neighbors* of the source node.
+
+
+Filtering returned nodes
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+In many cases, clients might not want to get all the traversed nodes in the
+response stream. With the ``return_nodes`` field (of type ``NodeFilter``), it
+is possible to specify various *criteria* for which nodes should be sent to the
+stream. By default, all nodes are returned.
+
+One common filter is to only want specific *node types* to be returned, which
+can be done with the ``types`` field of ``NodeFilter``. This field contains a
+node type restriction string (e.g. "dir,cnt,rev"), and defaults to "*" (all).
+For instance, to find the list of origins in which a given directory can be
+found:
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.Traverse \
+ "src: 'swh:1:dir:0000000000000000000000000000000000000006', return_nodes: {types: 'ori'}, direction: BACKWARD, mask: {paths: ['swhid']}"
+ swhid: "swh:1:ori:83404f995118bd25774f4ac14422a8f175e7a054"
+
+
+Traversal from multiple sources
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Traversals can have multiple starting nodes, when multiple source nodes are
+present in the ``src`` field. For instance, this BFS starts from two different
+directories, and explores the graph in parallel from these multiple starting
+points:
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.Traverse \
+ "src: ['swh:1:dir:0000000000000000000000000000000000000006', 'swh:1:dir:0000000000000000000000000000000000000017'], mask: {paths: ['swhid']}"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000006"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000017"
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000005"
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000004"
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000014"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000016"
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000015"
+
+
+Finding a path to a node matching a criteria
+--------------------------------------------
+
+The **FindPathTo** endpoint searches for a shortest path between a set of
+source nodes and any node that matches a specific *criteria*.
+It does so by performing a breadth-first search from the source node,
+until any node that matches the given criteria is found, then follows
+back its parents to return a shortest path from the source set to that
+node.
+
+The criteria can be specified in the ``target`` field of the
+``FindPathToRequest``, which is of type ``NodeFilter``.
+
+As an example, a common use-case for content provenance is to find the shortest
+path of a content to an origin in the transposed graph. This query can be
+run like this:
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.FindPathTo \
+ "src: 'swh:1:cnt:0000000000000000000000000000000000000001', target: {types: 'ori'}, direction: BACKWARD, mask: {paths: ['swhid']}"
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000001"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000008"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000009"
+ swhid: "swh:1:snp:0000000000000000000000000000000000000020"
+ swhid: "swh:1:ori:83404f995118bd25774f4ac14422a8f175e7a054"
+
+As soon as the request finds an origin, it stops and returns the path from the
+source set to this origin.
+
+Similar to the **Traverse** endpoint, it is possible to specify edge
+restrictions, graph directions, as well as multiple source nodes.
+
+
+Finding a path between two sets of nodes
+----------------------------------------
+
+The **FindPathBetween** endpoint searches for a shortest path between a set of
+source nodes and a set of destination nodes.
+
+It does so by performing a *bidirectional breadth-first search*, i.e.,
+two parallel breadth-first searches, one from the source set ("src-BFS")
+and one from the destination set ("dst-BFS"), until both searches find a
+common node that joins their visited sets. This node is called the
+"midpoint node".
+The path returned is the path src -> ... -> midpoint -> ... -> dst,
+which is always a shortest path between src and dst.
+
+The graph direction of both BFS can be configured separately. By
+default, the dst-BFS will use the graph in the opposite direction than
+the src-BFS (if direction = FORWARD, by default direction_reverse =
+BACKWARD, and vice-versa). The default behavior is thus to search for
+a shortest path between two nodes in a given direction. However, one
+can also specify FORWARD or BACKWARD for *both* the src-BFS and the
+dst-BFS. This will search for a common descendant or a common ancestor
+between the two sets, respectively. These will be the midpoints of the
+returned path.
+
+Similar to the **Traverse** endpoint, it is also possible to specify edge
+restrictions.
+
+**Example 1**: shortest path from a snapshot to a content (forward graph):
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.FindPathBetween \
+ "src: 'swh:1:snp:0000000000000000000000000000000000000020', dst: 'swh:1:cnt:0000000000000000000000000000000000000004', mask: {paths: ['swhid']}"
+ swhid: "swh:1:snp:0000000000000000000000000000000000000020"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000009"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000008"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000006"
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000004"
+
+**Example 2**: shortest path from a directory to a snapshot (backward graph):
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.FindPathBetween \
+ "src: 'swh:1:dir:0000000000000000000000000000000000000006', dst: 'swh:1:rel:0000000000000000000000000000000000000019', direction: BACKWARD, mask: {paths: ['swhid']}"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000006"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000008"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000012"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000013"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000018"
+ swhid: "swh:1:rel:0000000000000000000000000000000000000019"
+
+**Example 3**: common ancestor of two contents:
+
+.. code-block:: console
+
+ $ grpc_cli call localhost:50091 swh.graph.TraversalService.FindPathBetween \
+ "src: 'swh:1:cnt:0000000000000000000000000000000000000004', dst: 'swh:1:cnt:0000000000000000000000000000000000000015', direction: BACKWARD, direction_reverse: BACKWARD, mask: {paths: ['swhid']}"
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000004"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000006"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000008"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000012"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000013"
+ swhid: "swh:1:rev:0000000000000000000000000000000000000018"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000017"
+ swhid: "swh:1:dir:0000000000000000000000000000000000000016"
+ swhid: "swh:1:cnt:0000000000000000000000000000000000000015"
+ middle_node_index: 5
+
+Because ``middle_node_index = 5``, the common ancestor is
+``swh:1:rev:0000000000000000000000000000000000000018``.
+
+
+.. _swh-graph-grpc-api-protobuf:
+
+Protobuf API Reference
+======================
+
+The GRPC API is specified in a single self-documenting
+`protobuf <https://developers.google.com/protocol-buffers>`_ file, reproduced
+here verbatim.
+
+.. literalinclude:: ../proto/swhgraph.proto
+ :language: protobuf
diff --git a/docs/index.rst b/docs/index.rst
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -9,6 +9,7 @@
quickstart
api
+ grpc-api
java-api
memory
compression
diff --git a/java/README.md b/java/README.md
--- a/java/README.md
+++ b/java/README.md
@@ -15,13 +15,11 @@
```bash
$ java -cp target/swh-graph-*.jar \
- org.softwareheritage.graph.server.App \
+ org.softwareheritage.graph.rpc.GraphServer \
<compressed_graph_path>
```
-Default port is 5009 (use the `--port` option to change port number). If you
-need timings metadata send back to the client in addition to the result, use the
-`--timings` flag.
+Default port is 50091 (use the `--port` option to change port number).
Tests
-----
diff --git a/java/pom.xml b/java/pom.xml
--- a/java/pom.xml
+++ b/java/pom.xml
@@ -14,6 +14,8 @@
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.release>11</maven.compiler.release>
+ <protobuf.version>3.21.1</protobuf.version>
+ <grpc.version>1.47.0</grpc.version>
</properties>
<dependencies>
@@ -28,43 +30,17 @@
<version>5.7.0</version>
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>org.junit.vintage</groupId>
- <artifactId>junit-vintage-engine</artifactId>
- <version>5.7.0</version>
- </dependency>
- <dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
- <version>4.12</version>
- </dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.7.0</version>
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>org.hamcrest</groupId>
- <artifactId>hamcrest</artifactId>
- <version>2.2</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>io.javalin</groupId>
- <artifactId>javalin</artifactId>
- <version>3.0.0</version>
- </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.26</version>
</dependency>
- <dependency>
- <groupId>com.fasterxml.jackson.core</groupId>
- <artifactId>jackson-databind</artifactId>
- <version>2.13.0</version>
- </dependency>
<dependency>
<groupId>it.unimi.dsi</groupId>
<artifactId>webgraph-big</artifactId>
@@ -117,11 +93,6 @@
<artifactId>jsap</artifactId>
<version>2.1</version>
</dependency>
- <dependency>
- <groupId>net.sf.py4j</groupId>
- <artifactId>py4j</artifactId>
- <version>0.10.9.3</version>
- </dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
@@ -147,6 +118,46 @@
<artifactId>hadoop-client-runtime</artifactId>
<version>3.3.1</version>
</dependency>
+ <dependency>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java</artifactId>
+ <version>${protobuf.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-netty-shaded</artifactId>
+ <version>${grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-protobuf</artifactId>
+ <version>${grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-stub</artifactId>
+ <version>${grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-services</artifactId>
+ <version>${grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-testing</artifactId>
+ <version>${grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>javax.annotation</groupId>
+ <artifactId>javax.annotation-api</artifactId>
+ <version>1.3.2</version>
+ </dependency>
+ <dependency>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java-util</artifactId>
+ <version>${protobuf.version}</version>
+ </dependency>
</dependencies>
<build>
@@ -203,13 +214,17 @@
<artifactId>maven-project-info-reports-plugin</artifactId>
<version>3.0.0</version>
</plugin>
+ <plugin>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <version>3.1.2</version>
+ </plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<archive>
<manifest>
- <mainClass>org.softwareheritage.graph.server.App</mainClass>
+ <mainClass>org.softwareheritage.graph.rpc.GraphServer</mainClass>
</manifest>
</archive>
<descriptorRefs>
@@ -356,6 +371,33 @@
</tags>
</configuration>
</plugin>
+ <plugin>
+ <groupId>org.xolstice.maven.plugins</groupId>
+ <artifactId>protobuf-maven-plugin</artifactId>
+ <version>0.6.1</version>
+ <configuration>
+ <protocArtifact>com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
+ <pluginId>grpc-java</pluginId>
+ <pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}</pluginArtifact>
+ </configuration>
+ <executions>
+ <execution>
+ <goals>
+ <goal>compile</goal>
+ <goal>compile-custom</goal>
+ <goal>test-compile</goal>
+ <goal>test-compile-custom</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
</plugins>
+ <extensions>
+ <extension>
+ <groupId>kr.motd.maven</groupId>
+ <artifactId>os-maven-plugin</artifactId>
+ <version>1.6.2</version>
+ </extension>
+ </extensions>
</build>
</project>
diff --git a/java/src/main/java/org/softwareheritage/graph/AllowedEdges.java b/java/src/main/java/org/softwareheritage/graph/AllowedEdges.java
--- a/java/src/main/java/org/softwareheritage/graph/AllowedEdges.java
+++ b/java/src/main/java/org/softwareheritage/graph/AllowedEdges.java
@@ -71,4 +71,21 @@
return true;
return restrictedTo[srcType.ordinal()][dstType.ordinal()];
}
+
+ /**
+ * Return a new AllowedEdges instance with reversed edge restrictions. e.g. "src1:dst1,src2:dst2"
+ * becomes "dst1:src1,dst2:src2"
+ *
+ * @return a new AllowedEdges instance with reversed edge restrictions
+ */
+ public AllowedEdges reverse() {
+ AllowedEdges reversed = new AllowedEdges(null);
+ reversed.restrictedTo = new boolean[restrictedTo.length][restrictedTo[0].length];
+ for (int i = 0; i < restrictedTo.length; i++) {
+ for (int j = 0; j < restrictedTo[0].length; j++) {
+ reversed.restrictedTo[i][j] = restrictedTo[j][i];
+ }
+ }
+ return reversed;
+ }
}
diff --git a/java/src/main/java/org/softwareheritage/graph/Entry.java b/java/src/main/java/org/softwareheritage/graph/Entry.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/Entry.java
+++ /dev/null
@@ -1,193 +0,0 @@
-package org.softwareheritage.graph;
-
-import java.io.*;
-import java.util.ArrayList;
-
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.PropertyNamingStrategy;
-
-public class Entry {
- private SwhBidirectionalGraph graph;
-
- public void load_graph(String graphBasename) throws IOException {
- System.err.println("Loading graph " + graphBasename + " ...");
- this.graph = SwhBidirectionalGraph.loadMapped(graphBasename);
- System.err.println("Graph loaded.");
- }
-
- public SwhBidirectionalGraph get_graph() {
- return graph.copy();
- }
-
- public String stats() {
- try {
- Stats stats = new Stats(graph.getPath());
- ObjectMapper objectMapper = new ObjectMapper();
- objectMapper.setPropertyNamingStrategy(PropertyNamingStrategy.SNAKE_CASE);
- return objectMapper.writeValueAsString(stats);
- } catch (IOException e) {
- throw new RuntimeException("Cannot read stats: " + e);
- }
- }
-
- public void check_swhid(String src) {
- graph.getNodeId(new SWHID(src));
- }
-
- private int count_visitor(NodeCountVisitor f, long srcNodeId) {
- int[] count = {0};
- f.accept(srcNodeId, (node) -> {
- count[0]++;
- });
- return count[0];
- }
-
- public int count_leaves(String direction, String edgesFmt, String src, long maxEdges) {
- long srcNodeId = graph.getNodeId(new SWHID(src));
- Traversal t = new Traversal(graph.copy(), direction, edgesFmt, maxEdges);
- return count_visitor(t::leavesVisitor, srcNodeId);
- }
-
- public int count_neighbors(String direction, String edgesFmt, String src, long maxEdges) {
- long srcNodeId = graph.getNodeId(new SWHID(src));
- Traversal t = new Traversal(graph.copy(), direction, edgesFmt, maxEdges);
- return count_visitor(t::neighborsVisitor, srcNodeId);
- }
-
- public int count_visit_nodes(String direction, String edgesFmt, String src, long maxEdges) {
- long srcNodeId = graph.getNodeId(new SWHID(src));
- Traversal t = new Traversal(graph.copy(), direction, edgesFmt, maxEdges);
- return count_visitor(t::visitNodesVisitor, srcNodeId);
- }
-
- public QueryHandler get_handler(String clientFIFO) {
- return new QueryHandler(graph.copy(), clientFIFO);
- }
-
- private interface NodeCountVisitor {
- void accept(long nodeId, Traversal.NodeIdConsumer consumer);
- }
-
- public class QueryHandler {
- SwhBidirectionalGraph graph;
- BufferedWriter out;
- String clientFIFO;
-
- public QueryHandler(SwhBidirectionalGraph graph, String clientFIFO) {
- this.graph = graph;
- this.clientFIFO = clientFIFO;
- this.out = null;
- }
-
- public void writeNode(SWHID swhid) {
- try {
- out.write(swhid.toString() + "\n");
- } catch (IOException e) {
- throw new RuntimeException("Cannot write response to client: " + e);
- }
- }
-
- public void writeEdge(SWHID src, SWHID dst) {
- try {
- out.write(src.toString() + " " + dst.toString() + "\n");
- } catch (IOException e) {
- throw new RuntimeException("Cannot write response to client: " + e);
- }
- }
-
- public void open() {
- try {
- FileOutputStream file = new FileOutputStream(this.clientFIFO);
- this.out = new BufferedWriter(new OutputStreamWriter(file));
- } catch (IOException e) {
- throw new RuntimeException("Cannot open client FIFO: " + e);
- }
- }
-
- public void close() {
- try {
- out.close();
- } catch (IOException e) {
- throw new RuntimeException("Cannot write response to client: " + e);
- }
- }
-
- public void leaves(String direction, String edgesFmt, String src, long maxEdges, String returnTypes) {
- long srcNodeId = graph.getNodeId(new SWHID(src));
- open();
- Traversal t = new Traversal(graph, direction, edgesFmt, maxEdges, returnTypes);
- for (Long nodeId : t.leaves(srcNodeId)) {
- writeNode(graph.getSWHID(nodeId));
- }
- close();
- }
-
- public void neighbors(String direction, String edgesFmt, String src, long maxEdges, String returnTypes) {
- long srcNodeId = graph.getNodeId(new SWHID(src));
- open();
- Traversal t = new Traversal(graph, direction, edgesFmt, maxEdges, returnTypes);
- for (Long nodeId : t.neighbors(srcNodeId)) {
- writeNode(graph.getSWHID(nodeId));
- }
- close();
- }
-
- public void visit_nodes(String direction, String edgesFmt, String src, long maxEdges, String returnTypes) {
- long srcNodeId = graph.getNodeId(new SWHID(src));
- open();
- Traversal t = new Traversal(graph, direction, edgesFmt, maxEdges, returnTypes);
- for (Long nodeId : t.visitNodes(srcNodeId)) {
- writeNode(graph.getSWHID(nodeId));
- }
- close();
- }
-
- public void visit_edges(String direction, String edgesFmt, String src, long maxEdges, String returnTypes) {
- long srcNodeId = graph.getNodeId(new SWHID(src));
- open();
- Traversal t = new Traversal(graph, direction, edgesFmt, maxEdges);
- t.visitNodesVisitor(srcNodeId, null, (srcId, dstId) -> {
- writeEdge(graph.getSWHID(srcId), graph.getSWHID(dstId));
- });
- close();
- }
-
- public void walk(String direction, String edgesFmt, String algorithm, String src, String dst, long maxEdges,
- String returnTypes) {
- long srcNodeId = graph.getNodeId(new SWHID(src));
- open();
- ArrayList<Long> res;
- Traversal t = new Traversal(graph, direction, edgesFmt, maxEdges, returnTypes);
- if (dst.matches("ori|snp|rel|rev|dir|cnt")) {
- Node.Type dstType = Node.Type.fromStr(dst);
- res = t.walk(srcNodeId, dstType, algorithm);
- } else {
- long dstNodeId = graph.getNodeId(new SWHID(dst));
- res = t.walk(srcNodeId, dstNodeId, algorithm);
- }
- for (Long nodeId : res) {
- writeNode(graph.getSWHID(nodeId));
- }
- close();
- }
-
- public void random_walk(String direction, String edgesFmt, int retries, String src, String dst, long maxEdges,
- String returnTypes) {
- long srcNodeId = graph.getNodeId(new SWHID(src));
- open();
- ArrayList<Long> res;
- Traversal t = new Traversal(graph, direction, edgesFmt, maxEdges, returnTypes);
- if (dst.matches("ori|snp|rel|rev|dir|cnt")) {
- Node.Type dstType = Node.Type.fromStr(dst);
- res = t.randomWalk(srcNodeId, dstType, retries);
- } else {
- long dstNodeId = graph.getNodeId(new SWHID(dst));
- res = t.randomWalk(srcNodeId, dstNodeId, retries);
- }
- for (Long nodeId : res) {
- writeNode(graph.getSWHID(nodeId));
- }
- close();
- }
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/Stats.java b/java/src/main/java/org/softwareheritage/graph/Stats.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/Stats.java
+++ /dev/null
@@ -1,67 +0,0 @@
-package org.softwareheritage.graph;
-
-import java.io.FileInputStream;
-import java.io.IOException;
-import java.util.Properties;
-
-/**
- * Statistics on the compressed graph.
- * <p>
- * These statistics are not computed but directly read from
- * <a href="http://webgraph.di.unimi.it/">WebGraph</a> generated .stats and .properties files.
- *
- * @author The Software Heritage developers
- */
-
-public class Stats {
- public Counts counts;
- public Ratios ratios;
- public Degree indegree;
- public Degree outdegree;
- /**
- * Constructor.
- *
- * @param graphPath path and basename of compressed graph
- */
- public Stats(String graphPath) throws IOException {
- Properties properties = new Properties();
- properties.load(new FileInputStream(graphPath + ".properties"));
- properties.load(new FileInputStream(graphPath + ".stats"));
-
- this.counts = new Counts();
- this.ratios = new Ratios();
- this.indegree = new Degree();
- this.outdegree = new Degree();
-
- this.counts.nodes = Long.parseLong(properties.getProperty("nodes"));
- this.counts.edges = Long.parseLong(properties.getProperty("arcs"));
- this.ratios.compression = Double.parseDouble(properties.getProperty("compratio"));
- this.ratios.bitsPerNode = Double.parseDouble(properties.getProperty("bitspernode"));
- this.ratios.bitsPerEdge = Double.parseDouble(properties.getProperty("bitsperlink"));
- this.ratios.avgLocality = Double.parseDouble(properties.getProperty("avglocality"));
- this.indegree.min = Long.parseLong(properties.getProperty("minindegree"));
- this.indegree.max = Long.parseLong(properties.getProperty("maxindegree"));
- this.indegree.avg = Double.parseDouble(properties.getProperty("avgindegree"));
- this.outdegree.min = Long.parseLong(properties.getProperty("minoutdegree"));
- this.outdegree.max = Long.parseLong(properties.getProperty("maxoutdegree"));
- this.outdegree.avg = Double.parseDouble(properties.getProperty("avgoutdegree"));
- }
-
- public static class Counts {
- public long nodes;
- public long edges;
- }
-
- public static class Ratios {
- public double compression;
- public double bitsPerNode;
- public double bitsPerEdge;
- public double avgLocality;
- }
-
- public static class Degree {
- public long min;
- public long max;
- public double avg;
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/SwhBidirectionalGraph.java b/java/src/main/java/org/softwareheritage/graph/SwhBidirectionalGraph.java
--- a/java/src/main/java/org/softwareheritage/graph/SwhBidirectionalGraph.java
+++ b/java/src/main/java/org/softwareheritage/graph/SwhBidirectionalGraph.java
@@ -64,8 +64,8 @@
private SwhBidirectionalGraph(BidirectionalImmutableGraph graph, SwhGraphProperties properties) {
super(graph.forward, graph.backward);
- this.forwardGraph = (SwhUnidirectionalGraph) graph.forward;
- this.backwardGraph = (SwhUnidirectionalGraph) graph.backward;
+ this.forwardGraph = new SwhUnidirectionalGraph(graph.forward, properties);
+ this.backwardGraph = new SwhUnidirectionalGraph(graph.backward, properties);
this.properties = properties;
}
diff --git a/java/src/main/java/org/softwareheritage/graph/SwhGraph.java b/java/src/main/java/org/softwareheritage/graph/SwhGraph.java
--- a/java/src/main/java/org/softwareheritage/graph/SwhGraph.java
+++ b/java/src/main/java/org/softwareheritage/graph/SwhGraph.java
@@ -113,12 +113,12 @@
}
/** @see SwhGraphProperties#getMessage(long) */
- default byte[] getMessage(long nodeId) throws IOException {
+ default byte[] getMessage(long nodeId) {
return getProperties().getMessage(nodeId);
}
/** @see SwhGraphProperties#getUrl(long) */
- default String getUrl(long nodeId) throws IOException {
+ default String getUrl(long nodeId) {
return getProperties().getUrl(nodeId);
}
@@ -128,7 +128,7 @@
}
/** @see SwhGraphProperties#getTagName(long) */
- default byte[] getTagName(long nodeId) throws IOException {
+ default byte[] getTagName(long nodeId) {
return getProperties().getTagName(nodeId);
}
diff --git a/java/src/main/java/org/softwareheritage/graph/SwhGraphProperties.java b/java/src/main/java/org/softwareheritage/graph/SwhGraphProperties.java
--- a/java/src/main/java/org/softwareheritage/graph/SwhGraphProperties.java
+++ b/java/src/main/java/org/softwareheritage/graph/SwhGraphProperties.java
@@ -69,7 +69,6 @@
* Cleans up resources after use.
*/
public void close() throws IOException {
- nodeIdMap.close();
edgeLabelNames.close();
}
@@ -267,7 +266,7 @@
}
/** Get the message of the given revision or release node */
- public byte[] getMessage(long nodeId) throws IOException {
+ public byte[] getMessage(long nodeId) {
if (messageBuffer == null || messageOffsets == null) {
throw new IllegalStateException("Messages not loaded");
}
@@ -279,7 +278,7 @@
}
/** Get the URL of the given origin node */
- public String getUrl(long nodeId) throws IOException {
+ public String getUrl(long nodeId) {
byte[] url = getMessage(nodeId);
return (url != null) ? new String(url) : null;
}
@@ -291,7 +290,7 @@
}
/** Get the name of the given release node */
- public byte[] getTagName(long nodeId) throws IOException {
+ public byte[] getTagName(long nodeId) {
if (tagNameBuffer == null || tagNameOffsets == null) {
throw new IllegalStateException("Tag names not loaded");
}
diff --git a/java/src/main/java/org/softwareheritage/graph/SwhPath.java b/java/src/main/java/org/softwareheritage/graph/SwhPath.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/SwhPath.java
+++ /dev/null
@@ -1,122 +0,0 @@
-package org.softwareheritage.graph;
-
-import com.fasterxml.jackson.annotation.JsonValue;
-
-import java.util.ArrayList;
-
-/**
- * Wrapper class to store a list of {@link SWHID}.
- *
- * @author The Software Heritage developers
- * @see SWHID
- */
-
-public class SwhPath {
- /** Internal list of {@link SWHID} */
- ArrayList<SWHID> path;
-
- /**
- * Constructor.
- */
- public SwhPath() {
- this.path = new ArrayList<>();
- }
-
- /**
- * Constructor.
- *
- * @param swhids variable number of string SWHIDs to initialize this path with
- */
- public SwhPath(String... swhids) {
- this();
- for (String swhid : swhids) {
- add(new SWHID(swhid));
- }
- }
-
- /**
- * Constructor.
- *
- * @param swhids variable number of {@link SWHID} to initialize this path with
- * @see SWHID
- */
- public SwhPath(SWHID... swhids) {
- this();
- for (SWHID swhid : swhids) {
- add(swhid);
- }
- }
-
- /**
- * Returns this path as a list of {@link SWHID}.
- *
- * @return list of {@link SWHID} constituting the path
- * @see SWHID
- */
- @JsonValue
- public ArrayList<SWHID> getPath() {
- return path;
- }
-
- /**
- * Adds a {@link SWHID} to this path.
- *
- * @param swhid {@link SWHID} to add to this path
- * @see SWHID
- */
- public void add(SWHID swhid) {
- path.add(swhid);
- }
-
- /**
- * Returns the {@link SWHID} at the specified position in this path.
- *
- * @param index position of the {@link SWHID} to return
- * @return {@link SWHID} at the specified position
- * @see SWHID
- */
- public SWHID get(int index) {
- return path.get(index);
- }
-
- /**
- * Returns the number of elements in this path.
- *
- * @return number of elements in this path
- */
- public int size() {
- return path.size();
- }
-
- @Override
- public boolean equals(Object otherObj) {
- if (otherObj == this)
- return true;
- if (!(otherObj instanceof SwhPath))
- return false;
-
- SwhPath other = (SwhPath) otherObj;
- if (size() != other.size()) {
- return false;
- }
-
- for (int i = 0; i < size(); i++) {
- SWHID thisSWHID = get(i);
- SWHID otherSWHID = other.get(i);
- if (!thisSWHID.equals(otherSWHID)) {
- return false;
- }
- }
-
- return true;
- }
-
- @Override
- public String toString() {
- StringBuilder str = new StringBuilder();
- for (SWHID swhid : path) {
- str.append(swhid).append("/");
- }
- return str.toString();
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/SwhUnidirectionalGraph.java b/java/src/main/java/org/softwareheritage/graph/SwhUnidirectionalGraph.java
--- a/java/src/main/java/org/softwareheritage/graph/SwhUnidirectionalGraph.java
+++ b/java/src/main/java/org/softwareheritage/graph/SwhUnidirectionalGraph.java
@@ -34,7 +34,7 @@
/** Property data of the graph (id/type mappings etc.) */
public SwhGraphProperties properties;
- protected SwhUnidirectionalGraph(ImmutableGraph graph, SwhGraphProperties properties) {
+ public SwhUnidirectionalGraph(ImmutableGraph graph, SwhGraphProperties properties) {
this.graph = graph;
this.properties = properties;
}
diff --git a/java/src/main/java/org/softwareheritage/graph/Traversal.java b/java/src/main/java/org/softwareheritage/graph/Traversal.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/Traversal.java
+++ /dev/null
@@ -1,614 +0,0 @@
-package org.softwareheritage.graph;
-
-import java.util.ArrayDeque;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.LinkedList;
-import java.util.Map;
-import java.util.Queue;
-import java.util.Random;
-import java.util.Stack;
-import java.util.function.Consumer;
-import java.util.function.LongConsumer;
-
-import org.softwareheritage.graph.server.Endpoint;
-
-import it.unimi.dsi.big.webgraph.LazyLongIterator;
-
-/**
- * Traversal algorithms on the compressed graph.
- * <p>
- * Internal implementation of the traversal API endpoints. These methods only input/output internal
- * long ids, which are converted in the {@link Endpoint} higher-level class to {@link SWHID}.
- *
- * @author The Software Heritage developers
- * @see Endpoint
- */
-
-public class Traversal {
- /** Graph used in the traversal */
- SwhBidirectionalGraph graph;
- /** Type filter on the returned nodes */
- AllowedNodes nodesFilter;
- /** Restrictions on which edges can be traversed */
- AllowedEdges edgesRestrictions;
-
- /** Hash set storing if we have visited a node */
- HashSet<Long> visited;
- /** Hash map storing parent node id for each nodes during a traversal */
- Map<Long, Long> parentNode;
- /** Number of edges accessed during traversal */
- long nbEdgesAccessed;
-
- /** The anti Dos limit of edges traversed while a visit */
- long maxEdges;
-
- /** random number generator, for random walks */
- Random rng;
-
- /**
- * Constructor.
- *
- * @param graph graph used in the traversal
- * @param direction a string (either "forward" or "backward") specifying edge orientation
- * @param edgesFmt a formatted string describing <a href=
- * "https://docs.softwareheritage.org/devel/swh-graph/api.html#terminology">allowed
- * edges</a>
- */
-
- public Traversal(SwhBidirectionalGraph graph, String direction, String edgesFmt) {
- this(graph, direction, edgesFmt, 0);
- }
-
- public Traversal(SwhBidirectionalGraph graph, String direction, String edgesFmt, long maxEdges) {
- this(graph, direction, edgesFmt, maxEdges, "*");
- }
-
- public Traversal(SwhBidirectionalGraph graph, String direction, String edgesFmt, long maxEdges,
- String returnTypes) {
- if (!direction.matches("forward|backward")) {
- throw new IllegalArgumentException("Unknown traversal direction: " + direction);
- }
-
- if (direction.equals("backward")) {
- this.graph = graph.transpose();
- } else {
- this.graph = graph;
- }
- this.nodesFilter = new AllowedNodes(returnTypes);
- this.edgesRestrictions = new AllowedEdges(edgesFmt);
-
- this.visited = new HashSet<>();
- this.parentNode = new HashMap<>();
- this.nbEdgesAccessed = 0;
- this.maxEdges = maxEdges;
- this.rng = new Random();
- }
-
- /**
- * Returns number of accessed edges during traversal.
- *
- * @return number of edges accessed in last traversal
- */
- public long getNbEdgesAccessed() {
- return nbEdgesAccessed;
- }
-
- /**
- * Returns number of accessed nodes during traversal.
- *
- * @return number of nodes accessed in last traversal
- */
- public long getNbNodesAccessed() {
- return this.visited.size();
- }
-
- /**
- * Returns lazy iterator of successors of a node while following a specific set of edge types.
- *
- * @param g input graph
- * @param nodeId node specified as a long id
- * @param allowedEdges the specification of which edges can be traversed
- * @return lazy iterator of successors of the node, specified as a
- * <a href="http://webgraph.di.unimi.it/">WebGraph</a> LazyLongIterator
- */
- public static LazyLongIterator filterSuccessors(SwhBidirectionalGraph g, long nodeId, AllowedEdges allowedEdges) {
- if (allowedEdges.restrictedTo == null) {
- // All edges are allowed, bypass edge check
- return g.successors(nodeId);
- } else {
- LazyLongIterator allSuccessors = g.successors(nodeId);
- return new LazyLongIterator() {
- @Override
- public long nextLong() {
- long neighbor;
- while ((neighbor = allSuccessors.nextLong()) != -1) {
- if (allowedEdges.isAllowed(g.getNodeType(nodeId), g.getNodeType(neighbor))) {
- return neighbor;
- }
- }
- return -1;
- }
-
- @Override
- public long skip(final long n) {
- long i;
- for (i = 0; i < n && nextLong() != -1; i++)
- ;
- return i;
- }
- };
- }
- }
-
- private LazyLongIterator filterSuccessors(long nodeId, AllowedEdges allowedEdges) {
- return filterSuccessors(graph, nodeId, allowedEdges);
- }
-
- /**
- * Push version of {@link #leaves} will fire passed callback for each leaf.
- */
- public void leavesVisitor(long srcNodeId, NodeIdConsumer cb) {
- Stack<Long> stack = new Stack<>();
- this.nbEdgesAccessed = 0;
-
- stack.push(srcNodeId);
- visited.add(srcNodeId);
-
- while (!stack.isEmpty()) {
- long currentNodeId = stack.pop();
-
- long neighborsCnt = 0;
- nbEdgesAccessed += graph.outdegree(currentNodeId);
- if (this.maxEdges > 0) {
- if (nbEdgesAccessed >= this.maxEdges) {
- break;
- }
- }
- LazyLongIterator it = filterSuccessors(currentNodeId, edgesRestrictions);
- for (long neighborNodeId; (neighborNodeId = it.nextLong()) != -1;) {
- neighborsCnt++;
- if (!visited.contains(neighborNodeId)) {
- stack.push(neighborNodeId);
- visited.add(neighborNodeId);
- }
- }
-
- if (neighborsCnt == 0) {
- if (nodesFilter.isAllowed(graph.getNodeType(currentNodeId))) {
- cb.accept(currentNodeId);
- }
- }
- }
- }
-
- /**
- * Returns the leaves of a subgraph rooted at the specified source node.
- *
- * @param srcNodeId source node
- * @return list of node ids corresponding to the leaves
- */
- public ArrayList<Long> leaves(long srcNodeId) {
- ArrayList<Long> nodeIds = new ArrayList<Long>();
- leavesVisitor(srcNodeId, nodeIds::add);
- return nodeIds;
- }
-
- /**
- * Push version of {@link #neighbors}: will fire passed callback on each neighbor.
- */
- public void neighborsVisitor(long srcNodeId, NodeIdConsumer cb) {
- this.nbEdgesAccessed = graph.outdegree(srcNodeId);
- if (this.maxEdges > 0) {
- if (nbEdgesAccessed >= this.maxEdges) {
- return;
- }
- }
- LazyLongIterator it = filterSuccessors(srcNodeId, edgesRestrictions);
- for (long neighborNodeId; (neighborNodeId = it.nextLong()) != -1;) {
- if (nodesFilter.isAllowed(graph.getNodeType(neighborNodeId))) {
- cb.accept(neighborNodeId);
- }
- }
- }
-
- /**
- * Returns node direct neighbors (linked with exactly one edge).
- *
- * @param srcNodeId source node
- * @return list of node ids corresponding to the neighbors
- */
- public ArrayList<Long> neighbors(long srcNodeId) {
- ArrayList<Long> nodeIds = new ArrayList<>();
- neighborsVisitor(srcNodeId, nodeIds::add);
- return nodeIds;
- }
-
- /**
- * Push version of {@link #visitNodes}: will fire passed callback on each visited node.
- */
- public void visitNodesVisitor(long srcNodeId, NodeIdConsumer nodeCb, EdgeIdConsumer edgeCb) {
- Stack<Long> stack = new Stack<>();
- this.nbEdgesAccessed = 0;
-
- stack.push(srcNodeId);
- visited.add(srcNodeId);
-
- while (!stack.isEmpty()) {
- long currentNodeId = stack.pop();
- if (nodeCb != null) {
- if (nodesFilter.isAllowed(graph.getNodeType(currentNodeId))) {
- nodeCb.accept(currentNodeId);
- }
- }
- nbEdgesAccessed += graph.outdegree(currentNodeId);
- if (this.maxEdges > 0) {
- if (nbEdgesAccessed >= this.maxEdges) {
- break;
- }
- }
- LazyLongIterator it = filterSuccessors(currentNodeId, edgesRestrictions);
- for (long neighborNodeId; (neighborNodeId = it.nextLong()) != -1;) {
- if (edgeCb != null) {
- if (nodesFilter.isAllowed(graph.getNodeType(currentNodeId))) {
- edgeCb.accept(currentNodeId, neighborNodeId);
- }
- }
- if (!visited.contains(neighborNodeId)) {
- stack.push(neighborNodeId);
- visited.add(neighborNodeId);
- }
- }
- }
- }
-
- /** One-argument version to handle callbacks properly */
- public void visitNodesVisitor(long srcNodeId, NodeIdConsumer cb) {
- visitNodesVisitor(srcNodeId, cb, null);
- }
-
- /**
- * Performs a graph traversal and returns explored nodes.
- *
- * @param srcNodeId source node
- * @return list of explored node ids
- */
- public ArrayList<Long> visitNodes(long srcNodeId) {
- ArrayList<Long> nodeIds = new ArrayList<>();
- visitNodesVisitor(srcNodeId, nodeIds::add);
- return nodeIds;
- }
-
- /**
- * Push version of {@link #visitPaths}: will fire passed callback on each discovered (complete)
- * path.
- */
- public void visitPathsVisitor(long srcNodeId, PathConsumer cb) {
- Stack<Long> currentPath = new Stack<>();
- this.nbEdgesAccessed = 0;
- visitPathsInternalVisitor(srcNodeId, currentPath, cb);
- }
-
- /**
- * Performs a graph traversal and returns explored paths.
- *
- * @param srcNodeId source node
- * @return list of explored paths (represented as a list of node ids)
- */
- public ArrayList<ArrayList<Long>> visitPaths(long srcNodeId) {
- ArrayList<ArrayList<Long>> paths = new ArrayList<>();
- visitPathsVisitor(srcNodeId, paths::add);
- return paths;
- }
-
- private void visitPathsInternalVisitor(long currentNodeId, Stack<Long> currentPath, PathConsumer cb) {
- currentPath.push(currentNodeId);
-
- long visitedNeighbors = 0;
-
- nbEdgesAccessed += graph.outdegree(currentNodeId);
- if (this.maxEdges > 0) {
- if (nbEdgesAccessed >= this.maxEdges) {
- currentPath.pop();
- return;
- }
- }
- LazyLongIterator it = filterSuccessors(currentNodeId, edgesRestrictions);
- for (long neighborNodeId; (neighborNodeId = it.nextLong()) != -1;) {
- visitPathsInternalVisitor(neighborNodeId, currentPath, cb);
- visitedNeighbors++;
- }
-
- if (visitedNeighbors == 0) {
- ArrayList<Long> path = new ArrayList<>(currentPath);
- cb.accept(path);
- }
-
- currentPath.pop();
- }
-
- /**
- * Performs a graph traversal with backtracking, and returns the first found path from source to
- * destination.
- *
- * @param srcNodeId source node
- * @param dst destination (either a node or a node type)
- * @return found path as a list of node ids
- */
- public <T> ArrayList<Long> walk(long srcNodeId, T dst, String visitOrder) {
- long dstNodeId;
- if (visitOrder.equals("dfs")) {
- dstNodeId = walkInternalDFS(srcNodeId, dst);
- } else if (visitOrder.equals("bfs")) {
- dstNodeId = walkInternalBFS(srcNodeId, dst);
- } else {
- throw new IllegalArgumentException("Unknown visit order: " + visitOrder);
- }
-
- if (dstNodeId == -1) {
- throw new IllegalArgumentException("Cannot find destination: " + dst);
- }
-
- return backtracking(srcNodeId, dstNodeId);
- }
-
- /**
- * Performs a random walk (picking a random successor at each step) from source to destination.
- *
- * @param srcNodeId source node
- * @param dst destination (either a node or a node type)
- * @return found path as a list of node ids or an empty path to indicate that no suitable path have
- * been found
- */
- public <T> ArrayList<Long> randomWalk(long srcNodeId, T dst) {
- return randomWalk(srcNodeId, dst, 0);
- }
-
- /**
- * Performs a stubborn random walk (picking a random successor at each step) from source to
- * destination. The walk is "stubborn" in the sense that it will not give up the first time if a
- * satisfying target node is found, but it will retry up to a limited amount of times.
- *
- * @param srcNodeId source node
- * @param dst destination (either a node or a node type)
- * @param retries number of times to retry; 0 means no retries (single walk)
- * @return found path as a list of node ids or an empty path to indicate that no suitable path have
- * been found
- */
- public <T> ArrayList<Long> randomWalk(long srcNodeId, T dst, int retries) {
- long curNodeId = srcNodeId;
- ArrayList<Long> path = new ArrayList<>();
- this.nbEdgesAccessed = 0;
- boolean found;
-
- if (retries < 0) {
- throw new IllegalArgumentException("Negative number of retries given: " + retries);
- }
-
- while (true) {
- path.add(curNodeId);
- LazyLongIterator successors = filterSuccessors(curNodeId, edgesRestrictions);
- curNodeId = randomPick(successors);
- if (curNodeId < 0) {
- found = false;
- break;
- }
- if (isDstNode(curNodeId, dst)) {
- path.add(curNodeId);
- found = true;
- break;
- }
- }
-
- if (found) {
- return path;
- } else if (retries > 0) { // try again
- return randomWalk(srcNodeId, dst, retries - 1);
- } else { // not found and no retries left
- path.clear();
- return path;
- }
- }
-
- /**
- * Randomly choose an element from an iterator over Longs using reservoir sampling
- *
- * @param elements iterator over selection domain
- * @return randomly chosen element or -1 if no suitable element was found
- */
- private long randomPick(LazyLongIterator elements) {
- long curPick = -1;
- long seenCandidates = 0;
-
- for (long element; (element = elements.nextLong()) != -1;) {
- seenCandidates++;
- if (Math.round(rng.nextFloat() * (seenCandidates - 1)) == 0) {
- curPick = element;
- }
- }
-
- return curPick;
- }
-
- /**
- * Internal DFS function of {@link #walk}.
- *
- * @param srcNodeId source node
- * @param dst destination (either a node or a node type)
- * @return final destination node or -1 if no path found
- */
- private <T> long walkInternalDFS(long srcNodeId, T dst) {
- Stack<Long> stack = new Stack<>();
- this.nbEdgesAccessed = 0;
-
- stack.push(srcNodeId);
- visited.add(srcNodeId);
-
- while (!stack.isEmpty()) {
- long currentNodeId = stack.pop();
- if (isDstNode(currentNodeId, dst)) {
- return currentNodeId;
- }
-
- nbEdgesAccessed += graph.outdegree(currentNodeId);
- LazyLongIterator it = filterSuccessors(currentNodeId, edgesRestrictions);
- for (long neighborNodeId; (neighborNodeId = it.nextLong()) != -1;) {
- if (!visited.contains(neighborNodeId)) {
- stack.push(neighborNodeId);
- visited.add(neighborNodeId);
- parentNode.put(neighborNodeId, currentNodeId);
- }
- }
- }
-
- return -1;
- }
-
- /**
- * Internal BFS function of {@link #walk}.
- *
- * @param srcNodeId source node
- * @param dst destination (either a node or a node type)
- * @return final destination node or -1 if no path found
- */
- private <T> long walkInternalBFS(long srcNodeId, T dst) {
- Queue<Long> queue = new LinkedList<>();
- this.nbEdgesAccessed = 0;
-
- queue.add(srcNodeId);
- visited.add(srcNodeId);
-
- while (!queue.isEmpty()) {
- long currentNodeId = queue.poll();
- if (isDstNode(currentNodeId, dst)) {
- return currentNodeId;
- }
-
- nbEdgesAccessed += graph.outdegree(currentNodeId);
- LazyLongIterator it = filterSuccessors(currentNodeId, edgesRestrictions);
- for (long neighborNodeId; (neighborNodeId = it.nextLong()) != -1;) {
- if (!visited.contains(neighborNodeId)) {
- queue.add(neighborNodeId);
- visited.add(neighborNodeId);
- parentNode.put(neighborNodeId, currentNodeId);
- }
- }
- }
-
- return -1;
- }
-
- /**
- * Internal function of {@link #walk} to check if a node corresponds to the destination.
- *
- * @param nodeId current node
- * @param dst destination (either a node or a node type)
- * @return true if the node is a destination, or false otherwise
- */
- private <T> boolean isDstNode(long nodeId, T dst) {
- if (dst instanceof Long) {
- long dstNodeId = (Long) dst;
- return nodeId == dstNodeId;
- } else if (dst instanceof Node.Type) {
- Node.Type dstType = (Node.Type) dst;
- return graph.getNodeType(nodeId) == dstType;
- } else {
- return false;
- }
- }
-
- /**
- * Internal backtracking function of {@link #walk}.
- *
- * @param srcNodeId source node
- * @param dstNodeId destination node
- * @return the found path, as a list of node ids
- */
- private ArrayList<Long> backtracking(long srcNodeId, long dstNodeId) {
- ArrayList<Long> path = new ArrayList<>();
- long currentNodeId = dstNodeId;
- while (currentNodeId != srcNodeId) {
- path.add(currentNodeId);
- currentNodeId = parentNode.get(currentNodeId);
- }
- path.add(srcNodeId);
- Collections.reverse(path);
- return path;
- }
-
- /**
- * Find a common descendant between two given nodes using two parallel BFS
- *
- * @param lhsNode the first node
- * @param rhsNode the second node
- * @return the found path, as a list of node ids
- */
- public Long findCommonDescendant(long lhsNode, long rhsNode) {
- Queue<Long> lhsStack = new ArrayDeque<>();
- Queue<Long> rhsStack = new ArrayDeque<>();
- HashSet<Long> lhsVisited = new HashSet<>();
- HashSet<Long> rhsVisited = new HashSet<>();
- lhsStack.add(lhsNode);
- rhsStack.add(rhsNode);
- lhsVisited.add(lhsNode);
- rhsVisited.add(rhsNode);
-
- this.nbEdgesAccessed = 0;
- Long curNode;
-
- while (!lhsStack.isEmpty() || !rhsStack.isEmpty()) {
- if (!lhsStack.isEmpty()) {
- curNode = lhsStack.poll();
- nbEdgesAccessed += graph.outdegree(curNode);
- LazyLongIterator it = filterSuccessors(curNode, edgesRestrictions);
- for (long neighborNodeId; (neighborNodeId = it.nextLong()) != -1;) {
- if (!lhsVisited.contains(neighborNodeId)) {
- if (rhsVisited.contains(neighborNodeId))
- return neighborNodeId;
- lhsStack.add(neighborNodeId);
- lhsVisited.add(neighborNodeId);
- }
- }
- }
-
- if (!rhsStack.isEmpty()) {
- curNode = rhsStack.poll();
- nbEdgesAccessed += graph.outdegree(curNode);
- LazyLongIterator it = filterSuccessors(curNode, edgesRestrictions);
- for (long neighborNodeId; (neighborNodeId = it.nextLong()) != -1;) {
- if (!rhsVisited.contains(neighborNodeId)) {
- if (lhsVisited.contains(neighborNodeId))
- return neighborNodeId;
- rhsStack.add(neighborNodeId);
- rhsVisited.add(neighborNodeId);
- }
- }
- }
- }
-
- return null;
- }
-
- public interface NodeIdConsumer extends LongConsumer {
- /**
- * Callback for incrementally receiving node identifiers during a graph visit.
- */
- void accept(long nodeId);
- }
-
- public interface EdgeIdConsumer {
- /**
- * Callback for incrementally receiving edge identifiers during a graph visit.
- */
- void accept(long srcId, long dstId);
- }
-
- public interface PathConsumer extends Consumer<ArrayList<Long>> {
- /**
- * Callback for incrementally receiving node paths (made of node identifiers) during a graph visit.
- */
- void accept(ArrayList<Long> path);
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/benchmark/AccessEdge.java b/java/src/main/java/org/softwareheritage/graph/benchmark/AccessEdge.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/benchmark/AccessEdge.java
+++ /dev/null
@@ -1,45 +0,0 @@
-package org.softwareheritage.graph.benchmark;
-
-import com.martiansoftware.jsap.JSAPException;
-import it.unimi.dsi.big.webgraph.LazyLongIterator;
-import org.softwareheritage.graph.SwhBidirectionalGraph;
-import org.softwareheritage.graph.benchmark.utils.Statistics;
-import org.softwareheritage.graph.benchmark.utils.Timing;
-
-import java.io.IOException;
-import java.util.ArrayList;
-
-/**
- * Benchmark to time edge access time.
- *
- * @author The Software Heritage developers
- */
-
-public class AccessEdge {
- /**
- * Main entrypoint.
- *
- * @param args command line arguments
- */
- public static void main(String[] args) throws IOException, JSAPException {
- Benchmark bench = new Benchmark();
- bench.parseCommandLineArgs(args);
-
- SwhBidirectionalGraph graph = SwhBidirectionalGraph.loadMapped(bench.args.graphPath);
-
- long[] nodeIds = bench.args.random.generateNodeIds(graph, bench.args.nbNodes);
-
- ArrayList<Double> timings = new ArrayList<>();
- for (long nodeId : nodeIds) {
- long startTime = Timing.start();
- LazyLongIterator neighbors = graph.successors(nodeId);
- long firstNeighbor = neighbors.nextLong();
- double duration = Timing.stop(startTime);
- timings.add(duration);
- }
-
- System.out.println("Used " + bench.args.nbNodes + " random edges (results are in seconds):");
- Statistics stats = new Statistics(timings);
- stats.printAll();
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/benchmark/BFS.java b/java/src/main/java/org/softwareheritage/graph/benchmark/BFS.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/benchmark/BFS.java
+++ /dev/null
@@ -1,107 +0,0 @@
-package org.softwareheritage.graph.benchmark;
-
-import com.google.common.primitives.Longs;
-import com.martiansoftware.jsap.*;
-import it.unimi.dsi.big.webgraph.ImmutableGraph;
-import it.unimi.dsi.big.webgraph.LazyLongIterator;
-import it.unimi.dsi.bits.LongArrayBitVector;
-import it.unimi.dsi.fastutil.Arrays;
-import it.unimi.dsi.io.ByteDiskQueue;
-import it.unimi.dsi.logging.ProgressLogger;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.softwareheritage.graph.SwhBidirectionalGraph;
-
-import java.io.File;
-import java.io.IOException;
-
-public class BFS {
- private final static Logger LOGGER = LoggerFactory.getLogger(BFS.class);
- private final ImmutableGraph graph;
-
- public BFS(ImmutableGraph graph) {
- this.graph = graph;
- }
-
- private static JSAPResult parse_args(String[] args) {
- JSAPResult config = null;
- try {
- SimpleJSAP jsap = new SimpleJSAP(BFS.class.getName(), "",
- new Parameter[]{
- new FlaggedOption("graphPath", JSAP.STRING_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED, 'g',
- "graph", "Basename of the compressed graph"),
-
- new FlaggedOption("useTransposed", JSAP.BOOLEAN_PARSER, "false", JSAP.NOT_REQUIRED, 'T',
- "transposed", "Use transposed graph (default: false)"),});
-
- config = jsap.parse(args);
- if (jsap.messagePrinted()) {
- System.exit(1);
- }
- } catch (JSAPException e) {
- e.printStackTrace();
- }
- return config;
- }
-
- public static void main(String[] args) throws IOException {
- JSAPResult config = parse_args(args);
- String graphPath = config.getString("graphPath");
- boolean useTransposed = config.getBoolean("useTransposed");
-
- System.err.println("Loading graph " + graphPath + " ...");
- SwhBidirectionalGraph graph = SwhBidirectionalGraph.loadMapped(graphPath);
- System.err.println("Graph loaded.");
-
- if (useTransposed)
- graph = graph.transpose();
-
- BFS bfs = new BFS(graph);
- bfs.bfsperm();
- }
-
- // Partly inlined from it.unimi.dsi.law.big.graph.BFS
- private void bfsperm() throws IOException {
- final long n = graph.numNodes();
- // Allow enough memory to behave like in-memory queue
- int bufferSize = (int) Math.min(Arrays.MAX_ARRAY_SIZE & ~0x7, 8L * n);
-
- // Use a disk based queue to store BFS frontier
- final File queueFile = File.createTempFile(BFS.class.getSimpleName(), "queue");
- final ByteDiskQueue queue = ByteDiskQueue.createNew(queueFile, bufferSize, true);
- final byte[] byteBuf = new byte[Long.BYTES];
- // WARNING: no 64-bit version of this data-structure, but it can support
- // indices up to 2^37
- final LongArrayBitVector visited = LongArrayBitVector.ofLength(n);
- final ProgressLogger pl = new ProgressLogger(LOGGER);
- pl.expectedUpdates = n;
- pl.itemsName = "nodes";
- pl.start("Starting breadth-first visit...");
-
- for (long i = 0; i < n; i++) {
- if (visited.getBoolean(i))
- continue;
- queue.enqueue(Longs.toByteArray(i));
- visited.set(i);
-
- while (!queue.isEmpty()) {
- queue.dequeue(byteBuf);
- final long currentNode = Longs.fromByteArray(byteBuf);
-
- final LazyLongIterator iterator = graph.successors(currentNode);
- long succ;
- while ((succ = iterator.nextLong()) != -1) {
- if (!visited.getBoolean(succ)) {
- visited.set(succ);
- queue.enqueue(Longs.toByteArray(succ));
- }
- }
-
- pl.update();
- }
- }
-
- pl.done();
- queue.close();
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/benchmark/Benchmark.java b/java/src/main/java/org/softwareheritage/graph/benchmark/Benchmark.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/benchmark/Benchmark.java
+++ /dev/null
@@ -1,154 +0,0 @@
-package org.softwareheritage.graph.benchmark;
-
-import com.martiansoftware.jsap.*;
-import org.softwareheritage.graph.SwhBidirectionalGraph;
-import org.softwareheritage.graph.SWHID;
-import org.softwareheritage.graph.benchmark.utils.Random;
-import org.softwareheritage.graph.benchmark.utils.Statistics;
-import org.softwareheritage.graph.server.Endpoint;
-
-import java.io.BufferedWriter;
-import java.io.FileWriter;
-import java.io.IOException;
-import java.io.Writer;
-import java.util.ArrayList;
-import java.util.StringJoiner;
-import java.util.function.Function;
-
-/**
- * Benchmark common utility functions.
- *
- * @author The Software Heritage developers
- */
-
-public class Benchmark {
- /** CSV separator for log file */
- final String CSV_SEPARATOR = ";";
- /** Command line arguments */
- public Args args;
- /**
- * Constructor.
- */
- public Benchmark() {
- this.args = new Args();
- }
-
- /**
- * Parses benchmark command line arguments.
- *
- * @param args command line arguments
- */
- public void parseCommandLineArgs(String[] args) throws JSAPException {
- SimpleJSAP jsap = new SimpleJSAP(Benchmark.class.getName(),
- "Benchmark tool for Software Heritage use-cases scenarios.",
- new Parameter[]{
- new UnflaggedOption("graphPath", JSAP.STRING_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED,
- JSAP.NOT_GREEDY, "The basename of the compressed graph."),
- new FlaggedOption("nbNodes", JSAP.INTEGER_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED, 'n',
- "nb-nodes", "Number of random nodes used to do the benchmark."),
- new FlaggedOption("logFile", JSAP.STRING_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED, 'l',
- "log-file", "File name to output CSV format benchmark log."),
- new FlaggedOption("seed", JSAP.LONG_PARSER, JSAP.NO_DEFAULT, JSAP.NOT_REQUIRED, 's', "seed",
- "Random generator seed."),});
-
- JSAPResult config = jsap.parse(args);
- if (jsap.messagePrinted()) {
- System.exit(1);
- }
-
- this.args.graphPath = config.getString("graphPath");
- this.args.nbNodes = config.getInt("nbNodes");
- this.args.logFile = config.getString("logFile");
- this.args.random = config.contains("seed") ? new Random(config.getLong("seed")) : new Random();
- }
-
- /**
- * Creates CSV file for log output.
- */
- public void createCSVLogFile() throws IOException {
- try (Writer csvLog = new BufferedWriter(new FileWriter(args.logFile))) {
- StringJoiner csvHeader = new StringJoiner(CSV_SEPARATOR);
- csvHeader.add("use case name").add("SWHID").add("number of edges accessed").add("traversal timing")
- .add("swhid2node timing").add("node2swhid timing");
- csvLog.write(csvHeader.toString() + "\n");
- }
- }
-
- /**
- * Times a specific endpoint and outputs individual datapoints along with aggregated statistics.
- *
- * @param useCaseName benchmark use-case name
- * @param graph compressed graph used in the benchmark
- * @param nodeIds node ids to use as starting point for the endpoint traversal
- * @param operation endpoint function to benchmark
- * @param dstFmt destination formatted string as described in the
- * <a href="https://docs.softwareheritage.org/devel/swh-graph/api.html#walk">API</a>
- * @param algorithm traversal algorithm used in endpoint call (either "dfs" or "bfs")
- */
- public void timeEndpoint(String useCaseName, SwhBidirectionalGraph graph, long[] nodeIds,
- Function<Endpoint.Input, Endpoint.Output> operation, String dstFmt, String algorithm) throws IOException {
- ArrayList<Double> timings = new ArrayList<>();
- ArrayList<Double> timingsNormalized = new ArrayList<>();
- ArrayList<Double> nbEdgesAccessed = new ArrayList<>();
-
- final boolean append = true;
- try (Writer csvLog = new BufferedWriter(new FileWriter(args.logFile, append))) {
- for (long nodeId : nodeIds) {
- SWHID swhid = graph.getSWHID(nodeId);
-
- Endpoint.Output output = (dstFmt == null)
- ? operation.apply(new Endpoint.Input(swhid))
- : operation.apply(new Endpoint.Input(swhid, dstFmt, algorithm));
-
- StringJoiner csvLine = new StringJoiner(CSV_SEPARATOR);
- csvLine.add(useCaseName).add(swhid.toString()).add(Long.toString(output.meta.nbEdgesAccessed))
- .add(Double.toString(output.meta.timings.traversal))
- .add(Double.toString(output.meta.timings.swhid2node))
- .add(Double.toString(output.meta.timings.node2swhid));
- csvLog.write(csvLine.toString() + "\n");
-
- timings.add(output.meta.timings.traversal);
- nbEdgesAccessed.add((double) output.meta.nbEdgesAccessed);
- if (output.meta.nbEdgesAccessed != 0) {
- timingsNormalized.add(output.meta.timings.traversal / output.meta.nbEdgesAccessed);
- }
- }
- }
-
- System.out.println("\n" + useCaseName + " use-case:");
-
- System.out.println("timings:");
- Statistics stats = new Statistics(timings);
- stats.printAll();
-
- System.out.println("timings normalized:");
- Statistics statsNormalized = new Statistics(timingsNormalized);
- statsNormalized.printAll();
-
- System.out.println("nb edges accessed:");
- Statistics statsNbEdgesAccessed = new Statistics(nbEdgesAccessed);
- statsNbEdgesAccessed.printAll();
- }
-
- /**
- * Same as {@link #timeEndpoint} but without destination or algorithm specified to endpoint call.
- */
- public void timeEndpoint(String useCaseName, SwhBidirectionalGraph graph, long[] nodeIds,
- Function<Endpoint.Input, Endpoint.Output> operation) throws IOException {
- timeEndpoint(useCaseName, graph, nodeIds, operation, null, null);
- }
-
- /**
- * Input arguments.
- */
- public class Args {
- /** Basename of the compressed graph */
- public String graphPath;
- /** Number of random nodes to use for the benchmark */
- public int nbNodes;
- /** File name for CSV format benchmark log */
- public String logFile;
- /** Random generator */
- public Random random;
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/benchmark/Browsing.java b/java/src/main/java/org/softwareheritage/graph/benchmark/Browsing.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/benchmark/Browsing.java
+++ /dev/null
@@ -1,42 +0,0 @@
-package org.softwareheritage.graph.benchmark;
-
-import com.martiansoftware.jsap.JSAPException;
-import org.softwareheritage.graph.SwhBidirectionalGraph;
-import org.softwareheritage.graph.Node;
-import org.softwareheritage.graph.server.Endpoint;
-
-import java.io.IOException;
-
-/**
- * Benchmark Software Heritage
- * <a href="https://docs.softwareheritage.org/devel/swh-graph/use-cases.html#browsing">browsing
- * use-cases scenarios</a>.
- *
- * @author The Software Heritage developers
- */
-
-public class Browsing {
- /**
- * Main entrypoint.
- *
- * @param args command line arguments
- */
- public static void main(String[] args) throws IOException, JSAPException {
- Benchmark bench = new Benchmark();
- bench.parseCommandLineArgs(args);
-
- SwhBidirectionalGraph graph = SwhBidirectionalGraph.loadMapped(bench.args.graphPath);
-
- long[] dirNodeIds = bench.args.random.generateNodeIdsOfType(graph, bench.args.nbNodes, Node.Type.DIR);
- long[] revNodeIds = bench.args.random.generateNodeIdsOfType(graph, bench.args.nbNodes, Node.Type.REV);
-
- Endpoint dirEndpoint = new Endpoint(graph, "forward", "dir:cnt,dir:dir");
- Endpoint revEndpoint = new Endpoint(graph, "forward", "rev:rev");
-
- System.out.println("Used " + bench.args.nbNodes + " random nodes (results are in seconds):");
- bench.createCSVLogFile();
- bench.timeEndpoint("ls", graph, dirNodeIds, dirEndpoint::neighbors);
- bench.timeEndpoint("ls -R", graph, dirNodeIds, dirEndpoint::visitPaths);
- bench.timeEndpoint("git log", graph, revNodeIds, revEndpoint::visitNodes);
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/benchmark/Provenance.java b/java/src/main/java/org/softwareheritage/graph/benchmark/Provenance.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/benchmark/Provenance.java
+++ /dev/null
@@ -1,45 +0,0 @@
-package org.softwareheritage.graph.benchmark;
-
-import com.martiansoftware.jsap.JSAPException;
-import org.softwareheritage.graph.SwhBidirectionalGraph;
-import org.softwareheritage.graph.server.Endpoint;
-
-import java.io.IOException;
-
-/**
- * Benchmark Software Heritage
- * <a href="https://docs.softwareheritage.org/devel/swh-graph/use-cases.html#provenance">provenance
- * use-cases scenarios</a>.
- *
- * @author The Software Heritage developers
- */
-
-public class Provenance {
- /**
- * Main entrypoint.
- *
- * @param args command line arguments
- */
- public static void main(String[] args) throws IOException, JSAPException {
- Benchmark bench = new Benchmark();
- bench.parseCommandLineArgs(args);
-
- SwhBidirectionalGraph graph = SwhBidirectionalGraph.loadMapped(bench.args.graphPath);
-
- long[] nodeIds = bench.args.random.generateNodeIds(graph, bench.args.nbNodes);
-
- Endpoint commitProvenanceEndpoint = new Endpoint(graph, "backward", "dir:dir,cnt:dir,dir:rev");
- Endpoint originProvenanceEndpoint = new Endpoint(graph, "backward", "*");
-
- System.out.println("Used " + bench.args.nbNodes + " random nodes (results are in seconds):");
- bench.createCSVLogFile();
-
- bench.timeEndpoint("commit provenance (dfs)", graph, nodeIds, commitProvenanceEndpoint::walk, "rev", "dfs");
- bench.timeEndpoint("commit provenance (bfs)", graph, nodeIds, commitProvenanceEndpoint::walk, "rev", "bfs");
- bench.timeEndpoint("complete commit provenance", graph, nodeIds, commitProvenanceEndpoint::leaves);
-
- bench.timeEndpoint("origin provenance (dfs)", graph, nodeIds, originProvenanceEndpoint::walk, "ori", "dfs");
- bench.timeEndpoint("origin provenance (bfs)", graph, nodeIds, originProvenanceEndpoint::walk, "ori", "bfs");
- bench.timeEndpoint("complete origin provenance", graph, nodeIds, originProvenanceEndpoint::leaves);
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/benchmark/Vault.java b/java/src/main/java/org/softwareheritage/graph/benchmark/Vault.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/benchmark/Vault.java
+++ /dev/null
@@ -1,37 +0,0 @@
-package org.softwareheritage.graph.benchmark;
-
-import com.martiansoftware.jsap.JSAPException;
-import org.softwareheritage.graph.SwhBidirectionalGraph;
-import org.softwareheritage.graph.server.Endpoint;
-
-import java.io.IOException;
-
-/**
- * Benchmark Software Heritage
- * <a href="https://docs.softwareheritage.org/devel/swh-graph/use-cases.html#vault">vault use-case
- * scenario</a>.
- *
- * @author The Software Heritage developers
- */
-
-public class Vault {
- /**
- * Main entrypoint.
- *
- * @param args command line arguments
- */
- public static void main(String[] args) throws IOException, JSAPException {
- Benchmark bench = new Benchmark();
- bench.parseCommandLineArgs(args);
-
- SwhBidirectionalGraph graph = SwhBidirectionalGraph.loadMapped(bench.args.graphPath);
-
- long[] nodeIds = bench.args.random.generateNodeIds(graph, bench.args.nbNodes);
-
- Endpoint endpoint = new Endpoint(graph, "forward", "*");
-
- System.out.println("Used " + bench.args.nbNodes + " random nodes (results are in seconds):");
- bench.createCSVLogFile();
- bench.timeEndpoint("git bundle", graph, nodeIds, endpoint::visitNodes);
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/benchmark/utils/Random.java b/java/src/main/java/org/softwareheritage/graph/benchmark/utils/Random.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/benchmark/utils/Random.java
+++ /dev/null
@@ -1,67 +0,0 @@
-package org.softwareheritage.graph.benchmark.utils;
-
-import org.softwareheritage.graph.SwhBidirectionalGraph;
-import org.softwareheritage.graph.Node;
-
-import java.util.PrimitiveIterator;
-
-/**
- * Random related utility class.
- *
- * @author The Software Heritage developers
- */
-
-public class Random {
- /** Internal pseudorandom generator */
- java.util.Random random;
-
- /**
- * Constructor.
- */
- public Random() {
- this.random = new java.util.Random();
- }
-
- /**
- * Constructor.
- *
- * @param seed random generator seed
- */
- public Random(long seed) {
- this.random = new java.util.Random(seed);
- }
-
- /**
- * Generates random node ids.
- *
- * @param graph graph used to pick node ids
- * @param nbNodes number of node ids to generate
- * @return an array of random node ids
- */
- public long[] generateNodeIds(SwhBidirectionalGraph graph, int nbNodes) {
- return random.longs(nbNodes, 0, graph.numNodes()).toArray();
- }
-
- /**
- * Generates random node ids with a specific type.
- *
- * @param graph graph used to pick node ids
- * @param nbNodes number of node ids to generate
- * @param expectedType specific node type to pick
- * @return an array of random node ids
- */
- public long[] generateNodeIdsOfType(SwhBidirectionalGraph graph, int nbNodes, Node.Type expectedType) {
- PrimitiveIterator.OfLong nodes = random.longs(0, graph.numNodes()).iterator();
- long[] nodeIds = new long[nbNodes];
-
- long nextId;
- for (int i = 0; i < nbNodes; i++) {
- do {
- nextId = nodes.nextLong();
- } while (graph.getNodeType(nextId) != expectedType);
- nodeIds[i] = nextId;
- }
-
- return nodeIds;
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/benchmark/utils/Statistics.java b/java/src/main/java/org/softwareheritage/graph/benchmark/utils/Statistics.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/benchmark/utils/Statistics.java
+++ /dev/null
@@ -1,104 +0,0 @@
-package org.softwareheritage.graph.benchmark.utils;
-
-import java.util.ArrayList;
-import java.util.Collections;
-
-/**
- * Compute various statistics on a list of values.
- *
- * @author The Software Heritage developers
- */
-
-public class Statistics {
- /** Input values */
- ArrayList<Double> values;
-
- /**
- * Constructor.
- *
- * @param values input values
- */
- public Statistics(ArrayList<Double> values) {
- this.values = values;
- }
-
- /**
- * Returns the minimum value.
- *
- * @return minimum value
- */
- public double getMin() {
- double min = Double.POSITIVE_INFINITY;
- for (double v : values) {
- min = Math.min(min, v);
- }
- return min;
- }
-
- /**
- * Returns the maximum value.
- *
- * @return maximum value
- */
- public double getMax() {
- double max = Double.NEGATIVE_INFINITY;
- for (double v : values) {
- max = Math.max(max, v);
- }
- return max;
- }
-
- /**
- * Computes the average.
- *
- * @return average value
- */
- public double getAverage() {
- double sum = 0;
- for (double v : values) {
- sum += v;
- }
- return sum / (double) values.size();
- }
-
- /**
- * Returns the median value.
- *
- * @return median value
- */
- public double getMedian() {
- Collections.sort(values);
- int length = values.size();
- if (length % 2 == 0) {
- return (values.get(length / 2) + values.get(length / 2 - 1)) / 2;
- } else {
- return values.get(length / 2);
- }
- }
-
- /**
- * Computes the standard deviation.
- *
- * @return standard deviation value
- */
- public double getStandardDeviation() {
- double average = getAverage();
- double variance = 0;
- for (double v : values) {
- variance += (v - average) * (v - average);
- }
- variance /= (double) values.size();
- return Math.sqrt(variance);
- }
-
- /**
- * Computes and prints all statistical values.
- */
- public void printAll() {
- System.out.println("min value: " + getMin());
- System.out.println("max value: " + getMax());
- System.out.println("average: " + getAverage());
- System.out.println("median: " + getMedian());
- System.out.println("standard deviation: " + getStandardDeviation());
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/benchmark/utils/Timing.java b/java/src/main/java/org/softwareheritage/graph/benchmark/utils/Timing.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/benchmark/utils/Timing.java
+++ /dev/null
@@ -1,30 +0,0 @@
-package org.softwareheritage.graph.benchmark.utils;
-
-/**
- * Time measurement utility class.
- *
- * @author The Software Heritage developers
- */
-
-public class Timing {
- /**
- * Returns measurement starting timestamp.
- *
- * @return timestamp used for time measurement
- */
- public static long start() {
- return System.nanoTime();
- }
-
- /**
- * Ends timing measurement and returns total duration in seconds.
- *
- * @param startTime measurement starting timestamp
- * @return time in seconds elapsed since starting point
- */
- public static double stop(long startTime) {
- long endTime = System.nanoTime();
- double duration = (double) (endTime - startTime) / 1_000_000_000;
- return duration;
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/experiments/forks/FindCommonAncestor.java b/java/src/main/java/org/softwareheritage/graph/experiments/forks/FindCommonAncestor.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/experiments/forks/FindCommonAncestor.java
+++ /dev/null
@@ -1,62 +0,0 @@
-package org.softwareheritage.graph.experiments.forks;
-
-import com.martiansoftware.jsap.*;
-import org.softwareheritage.graph.SwhBidirectionalGraph;
-import org.softwareheritage.graph.Traversal;
-
-import java.io.IOException;
-import java.util.Scanner;
-
-public class FindCommonAncestor {
- private SwhBidirectionalGraph graph;
-
- private void load_graph(String graphBasename) throws IOException {
- System.err.println("Loading graph " + graphBasename + " ...");
- this.graph = SwhBidirectionalGraph.loadMapped(graphBasename);
- System.err.println("Graph loaded.");
- }
-
- private static JSAPResult parse_args(String[] args) {
- JSAPResult config = null;
- try {
- SimpleJSAP jsap = new SimpleJSAP(FindCommonAncestor.class.getName(), "",
- new Parameter[]{
- new FlaggedOption("edgesFmt", JSAP.STRING_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED, 'e',
- "edges", "Edges constraints"),
- new FlaggedOption("graphPath", JSAP.STRING_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED, 'g',
- "graph", "Basename of the compressed graph"),});
-
- config = jsap.parse(args);
- if (jsap.messagePrinted()) {
- System.exit(1);
- }
- } catch (JSAPException e) {
- e.printStackTrace();
- }
- return config;
- }
-
- public static void main(String[] args) {
- JSAPResult config = parse_args(args);
-
- String graphPath = config.getString("graphPath");
- String edgesFmt = config.getString("edgesFmt");
-
- FindCommonAncestor fca = new FindCommonAncestor();
- try {
- fca.load_graph(graphPath);
- } catch (IOException e) {
- System.out.println("Could not load graph: " + e);
- System.exit(2);
- }
-
- Scanner input = new Scanner(System.in);
- while (input.hasNextLong()) {
- long lhsNode = input.nextLong();
- long rhsNode = input.nextLong();
-
- Traversal t = new Traversal(fca.graph.symmetrize(), "forward", edgesFmt);
- System.out.println(t.findCommonDescendant(lhsNode, rhsNode));
- }
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/experiments/forks/FindPath.java b/java/src/main/java/org/softwareheritage/graph/experiments/forks/FindPath.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/experiments/forks/FindPath.java
+++ /dev/null
@@ -1,123 +0,0 @@
-package org.softwareheritage.graph.experiments.forks;
-
-import com.martiansoftware.jsap.*;
-import it.unimi.dsi.big.webgraph.LazyLongIterator;
-import org.softwareheritage.graph.SwhBidirectionalGraph;
-import org.softwareheritage.graph.Node;
-
-import java.io.IOException;
-import java.util.*;
-
-public class FindPath {
- private SwhBidirectionalGraph graph;
- private Long emptySnapshot;
-
- private void load_graph(String graphBasename) throws IOException {
- System.err.println("Loading graph " + graphBasename + " ...");
- this.graph = SwhBidirectionalGraph.loadMapped(graphBasename).symmetrize();
- System.err.println("Graph loaded.");
- this.emptySnapshot = null;
- }
-
- private static JSAPResult parse_args(String[] args) {
- JSAPResult config = null;
- try {
- SimpleJSAP jsap = new SimpleJSAP(FindPath.class.getName(), "",
- new Parameter[]{new FlaggedOption("graphPath", JSAP.STRING_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED,
- 'g', "graph", "Basename of the compressed graph"),});
-
- config = jsap.parse(args);
- if (jsap.messagePrinted()) {
- System.exit(1);
- }
- } catch (JSAPException e) {
- e.printStackTrace();
- }
- return config;
- }
-
- private boolean nodeIsEmptySnapshot(Long node) {
- if (this.emptySnapshot == null && this.graph.getNodeType(node) == Node.Type.SNP
- && this.graph.outdegree(node) == 0) {
- System.err.println("Found empty snapshot: " + node);
- this.emptySnapshot = node;
- }
- return node.equals(this.emptySnapshot);
- }
-
- private Boolean shouldVisit(Long node) {
- Node.Type nt = this.graph.getNodeType(node);
- if (nt != Node.Type.REV && nt != Node.Type.REL && nt != Node.Type.SNP && nt != Node.Type.ORI) {
- return false;
- }
- if (this.nodeIsEmptySnapshot(node))
- return false;
- return true;
- }
-
- private ArrayList<Long> findPath(Long src, Long dst) {
- HashSet<Long> visited = new HashSet<>();
- Queue<Long> queue = new ArrayDeque<>();
- Map<Long, Long> parentNode = new HashMap<>();
-
- queue.add(src);
- visited.add(src);
-
- while (!queue.isEmpty()) {
- long currentNode = queue.poll();
-
- final LazyLongIterator iterator = graph.successors(currentNode);
- long succ;
- while ((succ = iterator.nextLong()) != -1) {
- if (!shouldVisit(succ) || visited.contains(succ))
- continue;
- visited.add(succ);
- queue.add(succ);
- parentNode.put(succ, currentNode);
-
- if (succ == dst) {
- ArrayList<Long> path = new ArrayList<>();
- long n = dst;
- while (n != src) {
- path.add(n);
- n = parentNode.get(n);
- }
- path.add(src);
- Collections.reverse(path);
- return path;
- }
- }
- }
- return null;
- }
-
- public static void main(String[] args) {
- JSAPResult config = parse_args(args);
-
- String graphPath = config.getString("graphPath");
-
- FindPath fpath = new FindPath();
- try {
- fpath.load_graph(graphPath);
- } catch (IOException e) {
- System.out.println("Could not load graph: " + e);
- System.exit(2);
- }
-
- Scanner input = new Scanner(System.in);
- while (input.hasNextLong()) {
- long lhsNode = input.nextLong();
- long rhsNode = input.nextLong();
-
- ArrayList<Long> path = fpath.findPath(lhsNode, rhsNode);
- if (path != null) {
- for (Long n : path) {
- System.out.format("%d ", n);
- }
- System.out.println();
- } else {
- System.out.println("null");
- }
- }
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/experiments/multiplicationfactor/GenDistribution.java b/java/src/main/java/org/softwareheritage/graph/experiments/multiplicationfactor/GenDistribution.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/experiments/multiplicationfactor/GenDistribution.java
+++ /dev/null
@@ -1,130 +0,0 @@
-package org.softwareheritage.graph.experiments.multiplicationfactor;
-
-import com.martiansoftware.jsap.*;
-import org.softwareheritage.graph.SwhBidirectionalGraph;
-import org.softwareheritage.graph.Node;
-import org.softwareheritage.graph.Traversal;
-import org.softwareheritage.graph.benchmark.utils.Timing;
-
-import java.io.IOException;
-import java.util.Scanner;
-import java.util.concurrent.ArrayBlockingQueue;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-
-public class GenDistribution {
- private SwhBidirectionalGraph graph;
-
- private static JSAPResult parse_args(String[] args) {
- JSAPResult config = null;
- try {
- SimpleJSAP jsap = new SimpleJSAP(GenDistribution.class.getName(), "",
- new Parameter[]{
- new FlaggedOption("graphPath", JSAP.STRING_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED, 'g',
- "graph", "Basename of the compressed graph"),
- new FlaggedOption("srcType", JSAP.STRING_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED, 's',
- "srctype", "Source node type"),
- new FlaggedOption("dstType", JSAP.STRING_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED, 'd',
- "dsttype", "Destination node type"),
- new FlaggedOption("edgesFmt", JSAP.STRING_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED, 'e',
- "edges", "Edges constraints"),
-
- new FlaggedOption("numThreads", JSAP.INTEGER_PARSER, "128", JSAP.NOT_REQUIRED, 't',
- "numthreads", "Number of threads"),});
-
- config = jsap.parse(args);
- if (jsap.messagePrinted()) {
- System.exit(1);
- }
- } catch (JSAPException e) {
- e.printStackTrace();
- }
- return config;
- }
-
- public static void main(String[] args) {
- JSAPResult config = parse_args(args);
-
- String graphPath = config.getString("graphPath");
- Node.Type srcType = Node.Type.fromStr(config.getString("srcType"));
- Node.Type dstType = Node.Type.fromStr(config.getString("dstType"));
- String edgesFmt = config.getString("edgesFmt");
- int numThreads = config.getInt("numThreads");
-
- GenDistribution tp = new GenDistribution();
- try {
- tp.load_graph(graphPath);
- } catch (IOException e) {
- System.out.println("Could not load graph: " + e);
- System.exit(2);
- }
-
- final long END_OF_QUEUE = -1L;
-
- ArrayBlockingQueue<Long> queue = new ArrayBlockingQueue<>(numThreads);
- ExecutorService service = Executors.newFixedThreadPool(numThreads + 1);
-
- service.submit(() -> {
- try {
- Scanner input = new Scanner(System.in);
- while (input.hasNextLong()) {
- long node = input.nextLong();
- if (tp.graph.getNodeType(node) == srcType) {
- queue.put(node);
- }
- }
- } catch (InterruptedException e) {
- e.printStackTrace();
- } finally {
- for (int i = 0; i < numThreads; ++i) {
- try {
- queue.put(END_OF_QUEUE);
- } catch (InterruptedException e) {
- e.printStackTrace();
- }
- }
- }
- });
-
- for (int i = 0; i < numThreads; ++i) {
- service.submit(() -> {
- SwhBidirectionalGraph thread_graph = tp.graph.copy();
- long startTime;
- double totalTime;
-
- while (true) {
- Long node = null;
- try {
- node = queue.take();
- } catch (InterruptedException e) {
- e.printStackTrace();
- }
- if (node == null || node == END_OF_QUEUE) {
- return;
- }
-
- Traversal t = new Traversal(thread_graph, "backward", edgesFmt);
- int[] count = {0};
-
- startTime = Timing.start();
- t.visitNodesVisitor(node, (curnode) -> {
- if (tp.graph.getNodeType(curnode) == dstType) {
- count[0]++;
- }
- });
- totalTime = Timing.stop(startTime);
- System.out.format("%d %d %d %d %f\n", node, count[0], t.getNbNodesAccessed(),
- t.getNbEdgesAccessed(), totalTime);
- }
- });
- }
-
- service.shutdown();
- }
-
- private void load_graph(String graphBasename) throws IOException {
- System.err.println("Loading graph " + graphBasename + " ...");
- this.graph = SwhBidirectionalGraph.loadMapped(graphBasename);
- System.err.println("Graph loaded.");
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/maps/MapFile.java b/java/src/main/java/org/softwareheritage/graph/maps/MapFile.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/maps/MapFile.java
+++ /dev/null
@@ -1,66 +0,0 @@
-package org.softwareheritage.graph.maps;
-
-import it.unimi.dsi.io.ByteBufferInputStream;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.RandomAccessFile;
-import java.nio.channels.FileChannel;
-
-/**
- * Wrapper class around very big mmap()-ed file.
- * <p>
- * Java has a limit for mmap()-ed files because of unsupported 64-bit indexing. The
- * <a href="http://dsiutils.di.unimi.it/">dsiutils</a> ByteBufferInputStream is used to overcome
- * this Java limit.
- *
- * @author The Software Heritage developers
- */
-
-public class MapFile {
- /** Memory-mapped file buffer */
- ByteBufferInputStream bufferMap;
- /** Fixed line length of the mmap()-ed file */
- int lineLength;
-
- /**
- * Constructor.
- *
- * @param path file path to mmap()
- * @param lineLength fixed length of a line in the file
- */
- public MapFile(String path, int lineLength) throws IOException {
- this.bufferMap = null;
- this.lineLength = lineLength;
-
- try (RandomAccessFile mapFile = new RandomAccessFile(new File(path), "r")) {
- FileChannel fileChannel = mapFile.getChannel();
- bufferMap = ByteBufferInputStream.map(fileChannel, FileChannel.MapMode.READ_ONLY);
- }
- }
-
- /**
- * Returns a specific line in the file.
- *
- * @param lineIndex line number in the file
- * @return the line at the specified position
- */
- public byte[] readAtLine(long lineIndex) {
- byte[] buffer = new byte[lineLength];
- long position = lineIndex * (long) lineLength;
- bufferMap.position(position);
- bufferMap.read(buffer, 0, lineLength);
- return buffer;
- }
-
- public long size() {
- return bufferMap.length() / (long) lineLength;
- }
-
- /**
- * Closes the mmap()-ed file.
- */
- public void close() throws IOException {
- bufferMap.close();
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/maps/NodeIdMap.java b/java/src/main/java/org/softwareheritage/graph/maps/NodeIdMap.java
--- a/java/src/main/java/org/softwareheritage/graph/maps/NodeIdMap.java
+++ b/java/src/main/java/org/softwareheritage/graph/maps/NodeIdMap.java
@@ -1,17 +1,18 @@
package org.softwareheritage.graph.maps;
import it.unimi.dsi.fastutil.Size64;
+import it.unimi.dsi.fastutil.bytes.ByteBigList;
+import it.unimi.dsi.fastutil.bytes.ByteMappedBigList;
import it.unimi.dsi.fastutil.io.BinIO;
import it.unimi.dsi.fastutil.longs.LongBigList;
+import it.unimi.dsi.fastutil.longs.LongMappedBigList;
import it.unimi.dsi.fastutil.objects.Object2LongFunction;
-import it.unimi.dsi.util.ByteBufferLongBigList;
import org.softwareheritage.graph.SWHID;
import org.softwareheritage.graph.compress.NodeMapBuilder;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
-import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
/**
@@ -38,7 +39,7 @@
String graphPath;
/** mmap()-ed NODE_TO_SWHID file */
- MapFile nodeToSwhMap;
+ ByteBigList nodeToSwhMap;
/** Minimal perfect hash (MPH) function SWHID -> initial order */
Object2LongFunction<byte[]> mph;
@@ -54,14 +55,14 @@
this.graphPath = graphPath;
// node -> SWHID
- this.nodeToSwhMap = new MapFile(graphPath + NODE_TO_SWHID, SWHID_BIN_SIZE);
+ try (RandomAccessFile raf = new RandomAccessFile(graphPath + NODE_TO_SWHID, "r")) {
+ this.nodeToSwhMap = ByteMappedBigList.map(raf.getChannel());
+ }
// SWHID -> node
this.mph = loadMph(graphPath + ".mph");
-
try (RandomAccessFile mapFile = new RandomAccessFile(new File(graphPath + ".order"), "r")) {
- FileChannel fileChannel = mapFile.getChannel();
- this.orderMap = ByteBufferLongBigList.map(fileChannel);
+ this.orderMap = LongMappedBigList.map(mapFile.getChannel());
}
}
@@ -95,6 +96,7 @@
return legacyFunction.getLong(new String(bi, StandardCharsets.UTF_8));
}
+ @SuppressWarnings("deprecation")
@Override
public int size() {
return legacyFunction.size();
@@ -169,23 +171,19 @@
* Each line in NODE_TO_SWHID is formatted as: swhid The file is ordered by nodeId, meaning node0's
* swhid is at line 0, hence we can read the nodeId-th line to get corresponding swhid
*/
- if (nodeId < 0 || nodeId >= nodeToSwhMap.size()) {
- throw new IllegalArgumentException("Node id " + nodeId + " should be between 0 and " + nodeToSwhMap.size());
+ if (nodeId < 0 || nodeId >= nodeToSwhMap.size64()) {
+ throw new IllegalArgumentException(
+ "Node id " + nodeId + " should be between 0 and " + nodeToSwhMap.size64());
}
- return SWHID.fromBytes(nodeToSwhMap.readAtLine(nodeId));
- }
-
- /**
- * Closes the mapping files.
- */
- public void close() throws IOException {
- nodeToSwhMap.close();
+ byte[] swhid = new byte[SWHID_BIN_SIZE];
+ nodeToSwhMap.getElements(nodeId * SWHID_BIN_SIZE, swhid, 0, SWHID_BIN_SIZE);
+ return SWHID.fromBytes(swhid);
}
/** Return the number of nodes in the map. */
@Override
public long size64() {
- return nodeToSwhMap.size();
+ return nodeToSwhMap.size64();
}
}
diff --git a/java/src/main/java/org/softwareheritage/graph/rpc/GraphServer.java b/java/src/main/java/org/softwareheritage/graph/rpc/GraphServer.java
new file mode 100644
--- /dev/null
+++ b/java/src/main/java/org/softwareheritage/graph/rpc/GraphServer.java
@@ -0,0 +1,293 @@
+package org.softwareheritage.graph.rpc;
+
+import com.google.protobuf.FieldMask;
+import com.martiansoftware.jsap.*;
+import io.grpc.Server;
+import io.grpc.Status;
+import io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder;
+import io.grpc.netty.shaded.io.netty.channel.ChannelOption;
+import io.grpc.stub.StreamObserver;
+import io.grpc.protobuf.services.ProtoReflectionService;
+import it.unimi.dsi.logging.ProgressLogger;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.softwareheritage.graph.SWHID;
+import org.softwareheritage.graph.SwhBidirectionalGraph;
+import org.softwareheritage.graph.compress.LabelMapBuilder;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.util.Properties;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Server that manages startup/shutdown of a {@code Greeter} server.
+ */
+public class GraphServer {
+ private final static Logger logger = LoggerFactory.getLogger(GraphServer.class);
+
+ private final SwhBidirectionalGraph graph;
+ private final int port;
+ private final int threads;
+ private Server server;
+
+ /**
+ * @param graphBasename the basename of the SWH graph to load
+ * @param port the port on which the GRPC server will listen
+ * @param threads the number of threads to use in the server threadpool
+ */
+ public GraphServer(String graphBasename, int port, int threads) throws IOException {
+ this.graph = loadGraph(graphBasename);
+ this.port = port;
+ this.threads = threads;
+ }
+
+ /** Load a graph and all its properties. */
+ public static SwhBidirectionalGraph loadGraph(String basename) throws IOException {
+ // TODO: use loadLabelledMapped() when https://github.com/vigna/webgraph-big/pull/5 is merged
+ SwhBidirectionalGraph g = SwhBidirectionalGraph.loadLabelled(basename, new ProgressLogger(logger));
+ g.loadContentLength();
+ g.loadContentIsSkipped();
+ g.loadPersonIds();
+ g.loadAuthorTimestamps();
+ g.loadCommitterTimestamps();
+ g.loadMessages();
+ g.loadTagNames();
+ g.loadLabelNames();
+ return g;
+ }
+
+ /** Start the RPC server. */
+ private void start() throws IOException {
+ server = NettyServerBuilder.forPort(port).withChildOption(ChannelOption.SO_REUSEADDR, true)
+ .executor(Executors.newFixedThreadPool(threads)).addService(new TraversalService(graph))
+ .addService(ProtoReflectionService.newInstance()).build().start();
+ logger.info("Server started, listening on " + port);
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ try {
+ GraphServer.this.stop();
+ } catch (InterruptedException e) {
+ e.printStackTrace(System.err);
+ }
+ }));
+ }
+
+ private void stop() throws InterruptedException {
+ if (server != null) {
+ server.shutdown().awaitTermination(30, TimeUnit.SECONDS);
+ }
+ }
+
+ /**
+ * Await termination on the main thread since the grpc library uses daemon threads.
+ */
+ private void blockUntilShutdown() throws InterruptedException {
+ if (server != null) {
+ server.awaitTermination();
+ }
+ }
+
+ private static JSAPResult parseArgs(String[] args) {
+ JSAPResult config = null;
+ try {
+ SimpleJSAP jsap = new SimpleJSAP(LabelMapBuilder.class.getName(), "",
+ new Parameter[]{
+ new FlaggedOption("port", JSAP.INTEGER_PARSER, "50091", JSAP.NOT_REQUIRED, 'p', "port",
+ "The port on which the server should listen."),
+ new FlaggedOption("threads", JSAP.INTEGER_PARSER, "1", JSAP.NOT_REQUIRED, 't', "threads",
+ "The number of concurrent threads. 0 = number of cores."),
+ new UnflaggedOption("graphBasename", JSAP.STRING_PARSER, JSAP.REQUIRED,
+ "Basename of the output graph")});
+
+ config = jsap.parse(args);
+ if (jsap.messagePrinted()) {
+ System.exit(1);
+ }
+ } catch (JSAPException e) {
+ e.printStackTrace();
+ }
+ return config;
+ }
+
+ /** Main launches the server from the command line. */
+ public static void main(String[] args) throws IOException, InterruptedException {
+ JSAPResult config = parseArgs(args);
+ String graphBasename = config.getString("graphBasename");
+ int port = config.getInt("port");
+ int threads = config.getInt("threads");
+ if (threads == 0) {
+ threads = Runtime.getRuntime().availableProcessors();
+ }
+
+ final GraphServer server = new GraphServer(graphBasename, port, threads);
+ server.start();
+ server.blockUntilShutdown();
+ }
+
+ /** Implementation of the Traversal service, which contains all the graph querying endpoints. */
+ static class TraversalService extends TraversalServiceGrpc.TraversalServiceImplBase {
+ SwhBidirectionalGraph graph;
+
+ public TraversalService(SwhBidirectionalGraph graph) {
+ this.graph = graph;
+ }
+
+ /** Return various statistics on the overall graph. */
+ @Override
+ public void stats(StatsRequest request, StreamObserver<StatsResponse> responseObserver) {
+ StatsResponse.Builder response = StatsResponse.newBuilder();
+ response.setNumNodes(graph.numNodes());
+ response.setNumEdges(graph.numArcs());
+
+ Properties properties = new Properties();
+ try {
+ properties.load(new FileInputStream(graph.getPath() + ".properties"));
+ properties.load(new FileInputStream(graph.getPath() + ".stats"));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ response.setCompressionRatio(Double.parseDouble(properties.getProperty("compratio")));
+ response.setBitsPerNode(Double.parseDouble(properties.getProperty("bitspernode")));
+ response.setBitsPerEdge(Double.parseDouble(properties.getProperty("bitsperlink")));
+ response.setAvgLocality(Double.parseDouble(properties.getProperty("avglocality")));
+ response.setIndegreeMin(Long.parseLong(properties.getProperty("minindegree")));
+ response.setIndegreeMax(Long.parseLong(properties.getProperty("maxindegree")));
+ response.setIndegreeAvg(Double.parseDouble(properties.getProperty("avgindegree")));
+ response.setOutdegreeMin(Long.parseLong(properties.getProperty("minoutdegree")));
+ response.setOutdegreeMax(Long.parseLong(properties.getProperty("maxoutdegree")));
+ response.setOutdegreeAvg(Double.parseDouble(properties.getProperty("avgoutdegree")));
+ responseObserver.onNext(response.build());
+ responseObserver.onCompleted();
+ }
+
+ /** Return a single node and its properties. */
+ @Override
+ public void getNode(GetNodeRequest request, StreamObserver<Node> responseObserver) {
+ long nodeId;
+ try {
+ nodeId = graph.getNodeId(new SWHID(request.getSwhid()));
+ } catch (IllegalArgumentException e) {
+ responseObserver
+ .onError(Status.INVALID_ARGUMENT.withDescription(e.getMessage()).withCause(e).asException());
+ return;
+ }
+ Node.Builder builder = Node.newBuilder();
+ NodePropertyBuilder.buildNodeProperties(graph.getForwardGraph(),
+ request.hasMask() ? request.getMask() : null, builder, nodeId);
+ responseObserver.onNext(builder.build());
+ responseObserver.onCompleted();
+ }
+
+ /** Perform a BFS traversal from a set of source nodes and stream the nodes encountered. */
+ @Override
+ public void traverse(TraversalRequest request, StreamObserver<Node> responseObserver) {
+ SwhBidirectionalGraph g = graph.copy();
+ Traversal.SimpleTraversal t;
+ try {
+ t = new Traversal.SimpleTraversal(g, request, responseObserver::onNext);
+ } catch (IllegalArgumentException e) {
+ responseObserver
+ .onError(Status.INVALID_ARGUMENT.withDescription(e.getMessage()).withCause(e).asException());
+ return;
+ }
+ t.visit();
+ responseObserver.onCompleted();
+ }
+
+ /**
+ * Find the shortest path between a set of source nodes and a node that matches a given criteria
+ * using a BFS.
+ */
+ @Override
+ public void findPathTo(FindPathToRequest request, StreamObserver<Path> responseObserver) {
+ SwhBidirectionalGraph g = graph.copy();
+ Traversal.FindPathTo t;
+ try {
+ t = new Traversal.FindPathTo(g, request);
+ } catch (IllegalArgumentException e) {
+ responseObserver
+ .onError(Status.INVALID_ARGUMENT.withDescription(e.getMessage()).withCause(e).asException());
+ return;
+ }
+ t.visit();
+ Path path = t.getPath();
+ if (path == null) {
+ responseObserver.onError(Status.NOT_FOUND.asException());
+ } else {
+ responseObserver.onNext(path);
+ responseObserver.onCompleted();
+ }
+ }
+
+ /**
+ * Find the shortest path between a set of source nodes and a set of destination nodes using a
+ * bidirectional BFS.
+ */
+ @Override
+ public void findPathBetween(FindPathBetweenRequest request, StreamObserver<Path> responseObserver) {
+ SwhBidirectionalGraph g = graph.copy();
+ Traversal.FindPathBetween t;
+ try {
+ t = new Traversal.FindPathBetween(g, request);
+ } catch (IllegalArgumentException e) {
+ responseObserver
+ .onError(Status.INVALID_ARGUMENT.withDescription(e.getMessage()).withCause(e).asException());
+ return;
+ }
+ t.visit();
+ Path path = t.getPath();
+ if (path == null) {
+ responseObserver.onError(Status.NOT_FOUND.asException());
+ } else {
+ responseObserver.onNext(path);
+ responseObserver.onCompleted();
+ }
+ }
+
+ /** Return the number of nodes traversed by a BFS traversal. */
+ @Override
+ public void countNodes(TraversalRequest request, StreamObserver<CountResponse> responseObserver) {
+ AtomicLong count = new AtomicLong(0);
+ SwhBidirectionalGraph g = graph.copy();
+ TraversalRequest fixedReq = TraversalRequest.newBuilder(request)
+ // Ignore return fields, just count nodes
+ .setMask(FieldMask.getDefaultInstance()).build();
+ Traversal.SimpleTraversal t;
+ try {
+ t = new Traversal.SimpleTraversal(g, fixedReq, n -> count.incrementAndGet());
+ } catch (IllegalArgumentException e) {
+ responseObserver
+ .onError(Status.INVALID_ARGUMENT.withDescription(e.getMessage()).withCause(e).asException());
+ return;
+ }
+ t.visit();
+ CountResponse response = CountResponse.newBuilder().setCount(count.get()).build();
+ responseObserver.onNext(response);
+ responseObserver.onCompleted();
+ }
+
+ /** Return the number of edges traversed by a BFS traversal. */
+ @Override
+ public void countEdges(TraversalRequest request, StreamObserver<CountResponse> responseObserver) {
+ AtomicLong count = new AtomicLong(0);
+ SwhBidirectionalGraph g = graph.copy();
+ TraversalRequest fixedReq = TraversalRequest.newBuilder(request)
+ // Force return empty successors to count the edges
+ .setMask(FieldMask.newBuilder().addPaths("num_successors").build()).build();
+ Traversal.SimpleTraversal t;
+ try {
+ t = new Traversal.SimpleTraversal(g, fixedReq, n -> count.addAndGet(n.getNumSuccessors()));
+ } catch (IllegalArgumentException e) {
+ responseObserver
+ .onError(Status.INVALID_ARGUMENT.withDescription(e.getMessage()).withCause(e).asException());
+ return;
+ }
+ t.visit();
+ CountResponse response = CountResponse.newBuilder().setCount(count.get()).build();
+ responseObserver.onNext(response);
+ responseObserver.onCompleted();
+ }
+ }
+}
diff --git a/java/src/main/java/org/softwareheritage/graph/rpc/NodePropertyBuilder.java b/java/src/main/java/org/softwareheritage/graph/rpc/NodePropertyBuilder.java
new file mode 100644
--- /dev/null
+++ b/java/src/main/java/org/softwareheritage/graph/rpc/NodePropertyBuilder.java
@@ -0,0 +1,210 @@
+/*
+ * Copyright (c) 2022 The Software Heritage developers
+ * See the AUTHORS file at the top-level directory of this distribution
+ * License: GNU General Public License version 3, or any later version
+ * See top-level LICENSE file for more information
+ */
+
+package org.softwareheritage.graph.rpc;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.FieldMask;
+import com.google.protobuf.util.FieldMaskUtil;
+import it.unimi.dsi.big.webgraph.labelling.Label;
+import org.softwareheritage.graph.SwhUnidirectionalGraph;
+import org.softwareheritage.graph.labels.DirEntry;
+
+import java.util.*;
+
+/**
+ * NodePropertyBuilder is a helper class to enrich {@link Node} messages with node and edge
+ * properties. It is used by {@link GraphServer.TraversalService} to build the response messages or
+ * streams. Because property access is disk-based and slow, particular care is taken to avoid
+ * loading unnecessary properties. We use a FieldMask object to check which properties are requested
+ * by the client, and only load these.
+ */
+public class NodePropertyBuilder {
+ /**
+ * NodeDataMask caches a FieldMask into a more efficient representation (booleans). This avoids the
+ * need of parsing the FieldMask for each node in the stream.
+ */
+ public static class NodeDataMask {
+ public boolean swhid;
+ public boolean successor;
+ public boolean successorSwhid;
+ public boolean successorLabel;
+ public boolean numSuccessors;
+ public boolean cntLength;
+ public boolean cntIsSkipped;
+ public boolean revAuthor;
+ public boolean revAuthorDate;
+ public boolean revAuthorDateOffset;
+ public boolean revCommitter;
+ public boolean revCommitterDate;
+ public boolean revCommitterDateOffset;
+ public boolean revMessage;
+ public boolean relAuthor;
+ public boolean relAuthorDate;
+ public boolean relAuthorDateOffset;
+ public boolean relName;
+ public boolean relMessage;
+ public boolean oriUrl;
+
+ public NodeDataMask(FieldMask mask) {
+ Set<String> allowedFields = null;
+ if (mask != null) {
+ mask = FieldMaskUtil.normalize(mask);
+ allowedFields = new HashSet<>(mask.getPathsList());
+ }
+ this.swhid = allowedFields == null || allowedFields.contains("swhid");
+ this.successorSwhid = allowedFields == null || allowedFields.contains("successor")
+ || allowedFields.contains("successor.swhid");
+ this.successorLabel = allowedFields == null || allowedFields.contains("successor")
+ || allowedFields.contains("successor.label");
+ this.successor = this.successorSwhid || this.successorLabel;
+ this.numSuccessors = allowedFields == null || allowedFields.contains("num_successors");
+ this.cntLength = allowedFields == null || allowedFields.contains("cnt.length");
+ this.cntIsSkipped = allowedFields == null || allowedFields.contains("cnt.is_skipped");
+ this.revAuthor = allowedFields == null || allowedFields.contains("rev.author");
+ this.revAuthorDate = allowedFields == null || allowedFields.contains("rev.author_date");
+ this.revAuthorDateOffset = allowedFields == null || allowedFields.contains("rev.author_date_offset");
+ this.revCommitter = allowedFields == null || allowedFields.contains("rev.committer");
+ this.revCommitterDate = allowedFields == null || allowedFields.contains("rev.committer_date");
+ this.revCommitterDateOffset = allowedFields == null || allowedFields.contains("rev.committer_date_offset");
+ this.revMessage = allowedFields == null || allowedFields.contains("rev.message");
+ this.relAuthor = allowedFields == null || allowedFields.contains("rel.author");
+ this.relAuthorDate = allowedFields == null || allowedFields.contains("rel.author_date");
+ this.relAuthorDateOffset = allowedFields == null || allowedFields.contains("rel.author_date_offset");
+ this.relName = allowedFields == null || allowedFields.contains("rel.name");
+ this.relMessage = allowedFields == null || allowedFields.contains("rel.message");
+ this.oriUrl = allowedFields == null || allowedFields.contains("ori.url");
+ }
+ }
+
+ /** Enrich a Node message with node properties requested in the NodeDataMask. */
+ public static void buildNodeProperties(SwhUnidirectionalGraph graph, NodeDataMask mask, Node.Builder nodeBuilder,
+ long node) {
+ if (mask.swhid) {
+ nodeBuilder.setSwhid(graph.getSWHID(node).toString());
+ }
+
+ switch (graph.getNodeType(node)) {
+ case CNT:
+ ContentData.Builder cntBuilder = ContentData.newBuilder();
+ if (mask.cntLength) {
+ cntBuilder.setLength(graph.getContentLength(node));
+ }
+ if (mask.cntIsSkipped) {
+ cntBuilder.setIsSkipped(graph.isContentSkipped(node));
+ }
+ nodeBuilder.setCnt(cntBuilder.build());
+ break;
+ case REV:
+ RevisionData.Builder revBuilder = RevisionData.newBuilder();
+ if (mask.revAuthor) {
+ revBuilder.setAuthor(graph.getAuthorId(node));
+ }
+ if (mask.revAuthorDate) {
+ revBuilder.setAuthorDate(graph.getAuthorTimestamp(node));
+ }
+ if (mask.revAuthorDateOffset) {
+ revBuilder.setAuthorDateOffset(graph.getAuthorTimestampOffset(node));
+ }
+ if (mask.revCommitter) {
+ revBuilder.setCommitter(graph.getCommitterId(node));
+ }
+ if (mask.revCommitterDate) {
+ revBuilder.setCommitterDate(graph.getCommitterTimestamp(node));
+ }
+ if (mask.revCommitterDateOffset) {
+ revBuilder.setCommitterDateOffset(graph.getCommitterTimestampOffset(node));
+ }
+ if (mask.revMessage) {
+ byte[] msg = graph.getMessage(node);
+ if (msg != null) {
+ revBuilder.setMessage(ByteString.copyFrom(msg));
+ }
+ }
+ nodeBuilder.setRev(revBuilder.build());
+ break;
+ case REL:
+ ReleaseData.Builder relBuilder = ReleaseData.newBuilder();
+ if (mask.relAuthor) {
+ relBuilder.setAuthor(graph.getAuthorId(node));
+ }
+ if (mask.relAuthorDate) {
+ relBuilder.setAuthorDate(graph.getAuthorTimestamp(node));
+ }
+ if (mask.relAuthorDateOffset) {
+ relBuilder.setAuthorDateOffset(graph.getAuthorTimestampOffset(node));
+ }
+ if (mask.relName) {
+ byte[] msg = graph.getMessage(node);
+ if (msg != null) {
+ relBuilder.setMessage(ByteString.copyFrom(msg));
+ }
+ }
+ if (mask.relMessage) {
+ byte[] msg = graph.getMessage(node);
+ if (msg != null) {
+ relBuilder.setMessage(ByteString.copyFrom(msg));
+ }
+ }
+ nodeBuilder.setRel(relBuilder.build());
+ break;
+ case ORI:
+ OriginData.Builder oriBuilder = OriginData.newBuilder();
+ if (mask.oriUrl) {
+ String url = graph.getUrl(node);
+ if (url != null) {
+ oriBuilder.setUrl(url);
+ }
+ }
+ nodeBuilder.setOri(oriBuilder.build());
+ }
+ }
+
+ /** Enrich a Node message with node properties requested in the FieldMask. */
+ public static void buildNodeProperties(SwhUnidirectionalGraph graph, FieldMask mask, Node.Builder nodeBuilder,
+ long node) {
+ NodeDataMask nodeMask = new NodeDataMask(mask);
+ buildNodeProperties(graph, nodeMask, nodeBuilder, node);
+ }
+
+ /**
+ * Enrich a Node message with edge properties requested in the NodeDataMask, for a specific edge.
+ */
+ public static void buildSuccessorProperties(SwhUnidirectionalGraph graph, NodeDataMask mask,
+ Node.Builder nodeBuilder, long src, long dst, Label label) {
+ if (nodeBuilder != null) {
+ Successor.Builder successorBuilder = Successor.newBuilder();
+ if (mask.successorSwhid) {
+ successorBuilder.setSwhid(graph.getSWHID(dst).toString());
+ }
+ if (mask.successorLabel) {
+ DirEntry[] entries = (DirEntry[]) label.get();
+ for (DirEntry entry : entries) {
+ EdgeLabel.Builder builder = EdgeLabel.newBuilder();
+ builder.setName(ByteString.copyFrom(graph.getLabelName(entry.filenameId)));
+ builder.setPermission(entry.permission);
+ successorBuilder.addLabel(builder.build());
+ }
+ }
+ Successor successor = successorBuilder.build();
+ if (successor != Successor.getDefaultInstance()) {
+ nodeBuilder.addSuccessor(successor);
+ }
+
+ if (mask.numSuccessors) {
+ nodeBuilder.setNumSuccessors(nodeBuilder.getNumSuccessors() + 1);
+ }
+ }
+ }
+
+ /** Enrich a Node message with edge properties requested in the FieldMask, for a specific edge. */
+ public static void buildSuccessorProperties(SwhUnidirectionalGraph graph, FieldMask mask, Node.Builder nodeBuilder,
+ long src, long dst, Label label) {
+ NodeDataMask nodeMask = new NodeDataMask(mask);
+ buildSuccessorProperties(graph, nodeMask, nodeBuilder, src, dst, label);
+ }
+}
diff --git a/java/src/main/java/org/softwareheritage/graph/rpc/Traversal.java b/java/src/main/java/org/softwareheritage/graph/rpc/Traversal.java
new file mode 100644
--- /dev/null
+++ b/java/src/main/java/org/softwareheritage/graph/rpc/Traversal.java
@@ -0,0 +1,526 @@
+package org.softwareheritage.graph.rpc;
+
+import it.unimi.dsi.big.webgraph.labelling.ArcLabelledNodeIterator;
+import it.unimi.dsi.big.webgraph.labelling.Label;
+import org.softwareheritage.graph.*;
+
+import java.util.*;
+
+/** Traversal contains all the algorithms used for graph traversals */
+public class Traversal {
+ /**
+ * Wrapper around g.successors(), only follows edges that are allowed by the given
+ * {@link AllowedEdges} object.
+ */
+ private static ArcLabelledNodeIterator.LabelledArcIterator filterLabelledSuccessors(SwhUnidirectionalGraph g,
+ long nodeId, AllowedEdges allowedEdges) {
+ if (allowedEdges.restrictedTo == null) {
+ // All edges are allowed, bypass edge check
+ return g.labelledSuccessors(nodeId);
+ } else {
+ ArcLabelledNodeIterator.LabelledArcIterator allSuccessors = g.labelledSuccessors(nodeId);
+ return new ArcLabelledNodeIterator.LabelledArcIterator() {
+ @Override
+ public Label label() {
+ return allSuccessors.label();
+ }
+
+ @Override
+ public long nextLong() {
+ long neighbor;
+ while ((neighbor = allSuccessors.nextLong()) != -1) {
+ if (allowedEdges.isAllowed(g.getNodeType(nodeId), g.getNodeType(neighbor))) {
+ return neighbor;
+ }
+ }
+ return -1;
+ }
+
+ @Override
+ public long skip(final long n) {
+ long i = 0;
+ while (i < n && nextLong() != -1)
+ i++;
+ return i;
+ }
+ };
+ }
+ }
+
+ /** Helper class to check that a given node is "valid" for some given {@link NodeFilter} */
+ private static class NodeFilterChecker {
+ private final SwhUnidirectionalGraph g;
+ private final NodeFilter filter;
+ private final AllowedNodes allowedNodes;
+
+ private NodeFilterChecker(SwhUnidirectionalGraph graph, NodeFilter filter) {
+ this.g = graph;
+ this.filter = filter;
+ this.allowedNodes = new AllowedNodes(filter.hasTypes() ? filter.getTypes() : "*");
+ }
+
+ public boolean allowed(long nodeId) {
+ if (filter == null) {
+ return true;
+ }
+ if (!this.allowedNodes.isAllowed(g.getNodeType(nodeId))) {
+ return false;
+ }
+
+ return true;
+ }
+ }
+
+ /** Returns the unidirectional graph from a bidirectional graph and a {@link GraphDirection}. */
+ public static SwhUnidirectionalGraph getDirectedGraph(SwhBidirectionalGraph g, GraphDirection direction) {
+ switch (direction) {
+ case FORWARD:
+ return g.getForwardGraph();
+ case BACKWARD:
+ return g.getBackwardGraph();
+ /*
+ * TODO: add support for BOTH case BOTH: return new SwhUnidirectionalGraph(g.symmetrize(),
+ * g.getProperties());
+ */
+ default :
+ throw new IllegalArgumentException("Unknown direction: " + direction);
+ }
+ }
+
+ /** Returns the opposite of a given {@link GraphDirection} (equivalent to a graph transposition). */
+ public static GraphDirection reverseDirection(GraphDirection direction) {
+ switch (direction) {
+ case FORWARD:
+ return GraphDirection.BACKWARD;
+ case BACKWARD:
+ return GraphDirection.FORWARD;
+ /*
+ * TODO: add support for BOTH case BOTH: return GraphDirection.BOTH;
+ */
+ default :
+ throw new IllegalArgumentException("Unknown direction: " + direction);
+ }
+ }
+
+ /** Dummy exception to short-circuit and interrupt a graph traversal. */
+ static class StopTraversalException extends RuntimeException {
+ }
+
+ /** Generic BFS traversal algorithm. */
+ static class BFSVisitor {
+ /** The graph to traverse. */
+ protected final SwhUnidirectionalGraph g;
+ /** Depth of the node currently being visited */
+ protected long depth = 0;
+ /**
+ * Number of traversal successors (i.e., successors that will be considered by the traversal) of the
+ * node currently being visited
+ */
+ protected long traversalSuccessors = 0;
+ /** Number of edges accessed since the beginning of the traversal */
+ protected long edgesAccessed = 0;
+
+ /**
+ * Map from a node ID to its parent node ID. The key set can be used as the set of all visited
+ * nodes.
+ */
+ protected HashMap<Long, Long> parents = new HashMap<>();
+ /** Queue of nodes to visit (also called "frontier", "open set", "wavefront" etc.) */
+ protected ArrayDeque<Long> queue = new ArrayDeque<>();
+ /** If > 0, the maximum depth of the traversal. */
+ private long maxDepth = -1;
+ /** If > 0, the maximum number of edges to traverse. */
+ private long maxEdges = -1;
+
+ BFSVisitor(SwhUnidirectionalGraph g) {
+ this.g = g;
+ }
+
+ /** Add a new source node to the initial queue. */
+ public void addSource(long nodeId) {
+ queue.add(nodeId);
+ parents.put(nodeId, -1L);
+ }
+
+ /** Set the maximum depth of the traversal. */
+ public void setMaxDepth(long depth) {
+ maxDepth = depth;
+ }
+
+ /** Set the maximum number of edges to traverse. */
+ public void setMaxEdges(long edges) {
+ maxEdges = edges;
+ }
+
+ /** Setup the visit counters and depth sentinel. */
+ public void visitSetup() {
+ edgesAccessed = 0;
+ depth = 0;
+ queue.add(-1L); // depth sentinel
+ }
+
+ /** Perform the visit */
+ public void visit() {
+ visitSetup();
+ while (!queue.isEmpty()) {
+ visitStep();
+ }
+ }
+
+ /** Single "step" of a visit. Advance the frontier of exactly one node. */
+ public void visitStep() {
+ try {
+ assert !queue.isEmpty();
+ long curr = queue.poll();
+ if (curr == -1L) {
+ ++depth;
+ if (!queue.isEmpty()) {
+ queue.add(-1L);
+ visitStep();
+ }
+ return;
+ }
+ if (maxDepth >= 0 && depth > maxDepth) {
+ throw new StopTraversalException();
+ }
+ edgesAccessed += g.outdegree(curr);
+ if (maxEdges >= 0 && edgesAccessed > maxEdges) {
+ throw new StopTraversalException();
+ }
+ visitNode(curr);
+ } catch (StopTraversalException e) {
+ // Traversal is over, clear the to-do queue.
+ queue.clear();
+ }
+ }
+
+ /**
+ * Get the successors of a node. Override this function if you want to filter which successors are
+ * considered during the traversal.
+ */
+ protected ArcLabelledNodeIterator.LabelledArcIterator getSuccessors(long nodeId) {
+ return g.labelledSuccessors(nodeId);
+ }
+
+ /** Visit a node. Override to do additional processing on the node. */
+ protected void visitNode(long node) {
+ ArcLabelledNodeIterator.LabelledArcIterator it = getSuccessors(node);
+ traversalSuccessors = 0;
+ for (long succ; (succ = it.nextLong()) != -1;) {
+ traversalSuccessors++;
+ visitEdge(node, succ, it.label());
+ }
+ }
+
+ /** Visit an edge. Override to do additional processing on the edge. */
+ protected void visitEdge(long src, long dst, Label label) {
+ if (!parents.containsKey(dst)) {
+ queue.add(dst);
+ parents.put(dst, src);
+ }
+ }
+ }
+
+ /**
+ * SimpleTraversal is used by the Traverse endpoint. It extends BFSVisitor with additional
+ * processing, notably related to graph properties and filters.
+ */
+ static class SimpleTraversal extends BFSVisitor {
+ private final NodeFilterChecker nodeReturnChecker;
+ private final AllowedEdges allowedEdges;
+ private final TraversalRequest request;
+ private final NodePropertyBuilder.NodeDataMask nodeDataMask;
+ private final NodeObserver nodeObserver;
+
+ private Node.Builder nodeBuilder;
+
+ SimpleTraversal(SwhBidirectionalGraph bidirectionalGraph, TraversalRequest request, NodeObserver nodeObserver) {
+ super(getDirectedGraph(bidirectionalGraph, request.getDirection()));
+ this.request = request;
+ this.nodeObserver = nodeObserver;
+ this.nodeReturnChecker = new NodeFilterChecker(g, request.getReturnNodes());
+ this.nodeDataMask = new NodePropertyBuilder.NodeDataMask(request.hasMask() ? request.getMask() : null);
+ this.allowedEdges = new AllowedEdges(request.hasEdges() ? request.getEdges() : "*");
+ request.getSrcList().forEach(srcSwhid -> {
+ long srcNodeId = g.getNodeId(new SWHID(srcSwhid));
+ addSource(srcNodeId);
+ });
+ if (request.hasMaxDepth()) {
+ setMaxDepth(request.getMaxDepth());
+ }
+ if (request.hasMaxEdges()) {
+ setMaxEdges(request.getMaxEdges());
+ }
+ }
+
+ @Override
+ protected ArcLabelledNodeIterator.LabelledArcIterator getSuccessors(long nodeId) {
+ return filterLabelledSuccessors(g, nodeId, allowedEdges);
+ }
+
+ @Override
+ public void visitNode(long node) {
+ nodeBuilder = null;
+ if (nodeReturnChecker.allowed(node) && (!request.hasMinDepth() || depth >= request.getMinDepth())) {
+ nodeBuilder = Node.newBuilder();
+ NodePropertyBuilder.buildNodeProperties(g, nodeDataMask, nodeBuilder, node);
+ }
+ super.visitNode(node);
+ if (request.getReturnNodes().hasMinTraversalSuccessors()
+ && traversalSuccessors < request.getReturnNodes().getMinTraversalSuccessors()
+ || request.getReturnNodes().hasMaxTraversalSuccessors()
+ && traversalSuccessors > request.getReturnNodes().getMaxTraversalSuccessors()) {
+ nodeBuilder = null;
+ }
+ if (nodeBuilder != null) {
+ nodeObserver.onNext(nodeBuilder.build());
+ }
+ }
+
+ @Override
+ protected void visitEdge(long src, long dst, Label label) {
+ super.visitEdge(src, dst, label);
+ NodePropertyBuilder.buildSuccessorProperties(g, nodeDataMask, nodeBuilder, src, dst, label);
+ }
+ }
+
+ /**
+ * FindPathTo searches for a path from a source node to a node matching a given criteria It extends
+ * BFSVisitor with additional processing, and makes the traversal stop as soon as a node matching
+ * the given criteria is found.
+ */
+ static class FindPathTo extends BFSVisitor {
+ private final AllowedEdges allowedEdges;
+ private final FindPathToRequest request;
+ private final NodePropertyBuilder.NodeDataMask nodeDataMask;
+ private final NodeFilterChecker targetChecker;
+ private Long targetNode = null;
+
+ FindPathTo(SwhBidirectionalGraph bidirectionalGraph, FindPathToRequest request) {
+ super(getDirectedGraph(bidirectionalGraph, request.getDirection()));
+ this.request = request;
+ this.targetChecker = new NodeFilterChecker(g, request.getTarget());
+ this.nodeDataMask = new NodePropertyBuilder.NodeDataMask(request.hasMask() ? request.getMask() : null);
+ this.allowedEdges = new AllowedEdges(request.hasEdges() ? request.getEdges() : "*");
+ if (request.hasMaxDepth()) {
+ setMaxDepth(request.getMaxDepth());
+ }
+ if (request.hasMaxEdges()) {
+ setMaxEdges(request.getMaxEdges());
+ }
+ request.getSrcList().forEach(srcSwhid -> {
+ long srcNodeId = g.getNodeId(new SWHID(srcSwhid));
+ addSource(srcNodeId);
+ });
+ }
+
+ @Override
+ protected ArcLabelledNodeIterator.LabelledArcIterator getSuccessors(long nodeId) {
+ return filterLabelledSuccessors(g, nodeId, allowedEdges);
+ }
+
+ @Override
+ public void visitNode(long node) {
+ if (targetChecker.allowed(node)) {
+ targetNode = node;
+ throw new StopTraversalException();
+ }
+ super.visitNode(node);
+ }
+
+ /**
+ * Once the visit has been performed and a matching node has been found, return the shortest path
+ * from the source set to that node. To do so, we need to backtrack the parents of the node until we
+ * find one of the source nodes (whose parent is -1).
+ */
+ public Path getPath() {
+ if (targetNode == null) {
+ return null; // No path found.
+ }
+
+ /* Backtrack from targetNode to a source node */
+ long curNode = targetNode;
+ ArrayList<Long> path = new ArrayList<>();
+ while (curNode != -1) {
+ path.add(curNode);
+ curNode = parents.get(curNode);
+ }
+ Collections.reverse(path);
+
+ /* Enrich path with node properties */
+ Path.Builder pathBuilder = Path.newBuilder();
+ for (long nodeId : path) {
+ Node.Builder nodeBuilder = Node.newBuilder();
+ NodePropertyBuilder.buildNodeProperties(g, nodeDataMask, nodeBuilder, nodeId);
+ pathBuilder.addNode(nodeBuilder.build());
+ }
+ return pathBuilder.build();
+ }
+ }
+
+ /**
+ * FindPathBetween searches for a shortest path between a set of source nodes and a set of
+ * destination nodes.
+ *
+ * It does so by performing a *bidirectional breadth-first search*, i.e., two parallel breadth-first
+ * searches, one from the source set ("src-BFS") and one from the destination set ("dst-BFS"), until
+ * both searches find a common node that joins their visited sets. This node is called the "midpoint
+ * node". The path returned is the path src -> ... -> midpoint -> ... -> dst, which is always a
+ * shortest path between src and dst.
+ *
+ * The graph direction of both BFS can be configured separately. By default, the dst-BFS will use
+ * the graph in the opposite direction than the src-BFS (if direction = FORWARD, by default
+ * direction_reverse = BACKWARD, and vice-versa). The default behavior is thus to search for a
+ * shortest path between two nodes in a given direction. However, one can also specify FORWARD or
+ * BACKWARD for *both* the src-BFS and the dst-BFS. This will search for a common descendant or a
+ * common ancestor between the two sets, respectively. These will be the midpoints of the returned
+ * path.
+ */
+ static class FindPathBetween extends BFSVisitor {
+ private final FindPathBetweenRequest request;
+ private final NodePropertyBuilder.NodeDataMask nodeDataMask;
+ private final AllowedEdges allowedEdgesSrc;
+ private final AllowedEdges allowedEdgesDst;
+
+ private final BFSVisitor srcVisitor;
+ private final BFSVisitor dstVisitor;
+ private Long middleNode = null;
+
+ FindPathBetween(SwhBidirectionalGraph bidirectionalGraph, FindPathBetweenRequest request) {
+ super(getDirectedGraph(bidirectionalGraph, request.getDirection()));
+ this.request = request;
+ this.nodeDataMask = new NodePropertyBuilder.NodeDataMask(request.hasMask() ? request.getMask() : null);
+
+ GraphDirection direction = request.getDirection();
+ // if direction_reverse is not specified, use the opposite direction of direction
+ GraphDirection directionReverse = request.hasDirectionReverse()
+ ? request.getDirectionReverse()
+ : reverseDirection(request.getDirection());
+ SwhUnidirectionalGraph srcGraph = getDirectedGraph(bidirectionalGraph, direction);
+ SwhUnidirectionalGraph dstGraph = getDirectedGraph(bidirectionalGraph, directionReverse);
+
+ this.allowedEdgesSrc = new AllowedEdges(request.hasEdges() ? request.getEdges() : "*");
+ /*
+ * If edges_reverse is not specified: - If `edges` is not specified either, defaults to "*" - If
+ * direction == direction_reverse, defaults to `edges` - If direction != direction_reverse, defaults
+ * to the reverse of `edges` (e.g. "rev:dir" becomes "dir:rev").
+ */
+ this.allowedEdgesDst = request.hasEdgesReverse()
+ ? new AllowedEdges(request.getEdgesReverse())
+ : (request.hasEdges()
+ ? (direction == directionReverse
+ ? new AllowedEdges(request.getEdges())
+ : new AllowedEdges(request.getEdges()).reverse())
+ : new AllowedEdges("*"));
+
+ /*
+ * Source sub-visitor. Aborts as soon as it finds a node already visited by the destination
+ * sub-visitor.
+ */
+ this.srcVisitor = new BFSVisitor(srcGraph) {
+ @Override
+ protected ArcLabelledNodeIterator.LabelledArcIterator getSuccessors(long nodeId) {
+ return filterLabelledSuccessors(g, nodeId, allowedEdgesSrc);
+ }
+
+ @Override
+ public void visitNode(long node) {
+ if (dstVisitor.parents.containsKey(node)) {
+ middleNode = node;
+ throw new StopTraversalException();
+ }
+ super.visitNode(node);
+ }
+ };
+
+ /*
+ * Destination sub-visitor. Aborts as soon as it finds a node already visited by the source
+ * sub-visitor.
+ */
+ this.dstVisitor = new BFSVisitor(dstGraph) {
+ @Override
+ protected ArcLabelledNodeIterator.LabelledArcIterator getSuccessors(long nodeId) {
+ return filterLabelledSuccessors(g, nodeId, allowedEdgesDst);
+ }
+
+ @Override
+ public void visitNode(long node) {
+ if (srcVisitor.parents.containsKey(node)) {
+ middleNode = node;
+ throw new StopTraversalException();
+ }
+ super.visitNode(node);
+ }
+ };
+ if (request.hasMaxDepth()) {
+ this.srcVisitor.setMaxDepth(request.getMaxDepth());
+ this.dstVisitor.setMaxDepth(request.getMaxDepth());
+ }
+ if (request.hasMaxEdges()) {
+ this.srcVisitor.setMaxEdges(request.getMaxEdges());
+ this.dstVisitor.setMaxEdges(request.getMaxEdges());
+ }
+ request.getSrcList().forEach(srcSwhid -> {
+ long srcNodeId = g.getNodeId(new SWHID(srcSwhid));
+ srcVisitor.addSource(srcNodeId);
+ });
+ request.getDstList().forEach(srcSwhid -> {
+ long srcNodeId = g.getNodeId(new SWHID(srcSwhid));
+ dstVisitor.addSource(srcNodeId);
+ });
+ }
+
+ @Override
+ public void visit() {
+ /*
+ * Bidirectional BFS: maintain two sub-visitors, and alternately run a visit step in each of them.
+ */
+ srcVisitor.visitSetup();
+ dstVisitor.visitSetup();
+ while (!srcVisitor.queue.isEmpty() || !dstVisitor.queue.isEmpty()) {
+ if (!srcVisitor.queue.isEmpty()) {
+ srcVisitor.visitStep();
+ }
+ if (!dstVisitor.queue.isEmpty()) {
+ dstVisitor.visitStep();
+ }
+ }
+ }
+
+ public Path getPath() {
+ if (middleNode == null) {
+ return null; // No path found.
+ }
+ Path.Builder pathBuilder = Path.newBuilder();
+ ArrayList<Long> path = new ArrayList<>();
+
+ /* First section of the path: src -> midpoint */
+ long curNode = middleNode;
+ while (curNode != -1) {
+ path.add(curNode);
+ curNode = srcVisitor.parents.get(curNode);
+ }
+ pathBuilder.setMidpointIndex(path.size() - 1);
+ Collections.reverse(path);
+
+ /* Second section of the path: midpoint -> dst */
+ curNode = dstVisitor.parents.get(middleNode);
+ while (curNode != -1) {
+ path.add(curNode);
+ curNode = dstVisitor.parents.get(curNode);
+ }
+
+ /* Enrich path with node properties */
+ for (long nodeId : path) {
+ Node.Builder nodeBuilder = Node.newBuilder();
+ NodePropertyBuilder.buildNodeProperties(g, nodeDataMask, nodeBuilder, nodeId);
+ pathBuilder.addNode(nodeBuilder.build());
+ }
+ return pathBuilder.build();
+ }
+ }
+
+ public interface NodeObserver {
+ void onNext(Node nodeId);
+ }
+}
diff --git a/java/src/main/java/org/softwareheritage/graph/server/App.java b/java/src/main/java/org/softwareheritage/graph/server/App.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/server/App.java
+++ /dev/null
@@ -1,196 +0,0 @@
-package org.softwareheritage.graph.server;
-
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.PropertyNamingStrategy;
-import com.martiansoftware.jsap.*;
-import io.javalin.Javalin;
-import io.javalin.http.Context;
-import io.javalin.plugin.json.JavalinJackson;
-import org.softwareheritage.graph.SwhBidirectionalGraph;
-import org.softwareheritage.graph.Stats;
-import org.softwareheritage.graph.SWHID;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.Map;
-
-/**
- * Web framework of the swh-graph server RPC API.
- *
- * @author The Software Heritage developers
- */
-
-public class App {
- /**
- * Main entrypoint.
- *
- * @param args command line arguments
- */
- public static void main(String[] args) throws IOException, JSAPException {
- SimpleJSAP jsap = new SimpleJSAP(App.class.getName(),
- "Server to load and query a compressed graph representation of Software Heritage archive.",
- new Parameter[]{
- new FlaggedOption("port", JSAP.INTEGER_PARSER, "5009", JSAP.NOT_REQUIRED, 'p', "port",
- "Binding port of the server."),
- new UnflaggedOption("graphPath", JSAP.STRING_PARSER, JSAP.NO_DEFAULT, JSAP.REQUIRED,
- JSAP.NOT_GREEDY, "The basename of the compressed graph."),
- new Switch("timings", 't', "timings", "Show timings in API result metadata."),});
-
- JSAPResult config = jsap.parse(args);
- if (jsap.messagePrinted()) {
- System.exit(1);
- }
-
- String graphPath = config.getString("graphPath");
- int port = config.getInt("port");
- boolean showTimings = config.getBoolean("timings");
-
- startServer(graphPath, port, showTimings);
- }
-
- /**
- * Loads compressed graph and starts the web server to query it.
- *
- * @param graphPath basename of the compressed graph
- * @param port binding port of the server
- * @param showTimings true if timings should be in results metadata, false otherwise
- */
- private static void startServer(String graphPath, int port, boolean showTimings) throws IOException {
- SwhBidirectionalGraph graph = SwhBidirectionalGraph.loadMapped(graphPath);
- Stats stats = new Stats(graphPath);
-
- // Clean up on exit
- Runtime.getRuntime().addShutdownHook(new Thread() {
- public void run() {
- try {
- graph.close();
- } catch (IOException e) {
- System.out.println("Could not clean up graph on exit: " + e);
- }
- }
- });
-
- // Configure Jackson JSON to use snake case naming style
- ObjectMapper objectMapper = JavalinJackson.getObjectMapper();
- objectMapper.setPropertyNamingStrategy(PropertyNamingStrategy.SNAKE_CASE);
- JavalinJackson.configure(objectMapper);
-
- Javalin app = Javalin.create().start(port);
-
- app.before("/stats/*", ctx -> {
- checkQueryStrings(ctx, "");
- });
- app.before("/leaves/*", ctx -> {
- checkQueryStrings(ctx, "direction|edges");
- });
- app.before("/neighbors/*", ctx -> {
- checkQueryStrings(ctx, "direction|edges");
- });
- app.before("/visit/*", ctx -> {
- checkQueryStrings(ctx, "direction|edges");
- });
- app.before("/walk/*", ctx -> {
- checkQueryStrings(ctx, "direction|edges|traversal");
- });
-
- app.get("/stats/", ctx -> {
- ctx.json(stats);
- });
-
- // Graph traversal endpoints
- // By default the traversal is a forward DFS using all edges
-
- app.get("/leaves/:src", ctx -> {
- SWHID src = new SWHID(ctx.pathParam("src"));
- String direction = ctx.queryParam("direction", "forward");
- String edgesFmt = ctx.queryParam("edges", "*");
-
- Endpoint endpoint = new Endpoint(graph, direction, edgesFmt);
- Endpoint.Output output = endpoint.leaves(new Endpoint.Input(src));
- ctx.json(formatEndpointOutput(output, showTimings));
- });
-
- app.get("/neighbors/:src", ctx -> {
- SWHID src = new SWHID(ctx.pathParam("src"));
- String direction = ctx.queryParam("direction", "forward");
- String edgesFmt = ctx.queryParam("edges", "*");
-
- Endpoint endpoint = new Endpoint(graph, direction, edgesFmt);
- Endpoint.Output output = endpoint.neighbors(new Endpoint.Input(src));
- ctx.json(formatEndpointOutput(output, showTimings));
- });
-
- app.get("/visit/nodes/:src", ctx -> {
- SWHID src = new SWHID(ctx.pathParam("src"));
- String direction = ctx.queryParam("direction", "forward");
- String edgesFmt = ctx.queryParam("edges", "*");
-
- Endpoint endpoint = new Endpoint(graph, direction, edgesFmt);
- Endpoint.Output output = endpoint.visitNodes(new Endpoint.Input(src));
- ctx.json(formatEndpointOutput(output, showTimings));
- });
-
- app.get("/visit/paths/:src", ctx -> {
- SWHID src = new SWHID(ctx.pathParam("src"));
- String direction = ctx.queryParam("direction", "forward");
- String edgesFmt = ctx.queryParam("edges", "*");
-
- Endpoint endpoint = new Endpoint(graph, direction, edgesFmt);
- Endpoint.Output output = endpoint.visitPaths(new Endpoint.Input(src));
- ctx.json(formatEndpointOutput(output, showTimings));
- });
-
- app.get("/walk/:src/:dst", ctx -> {
- SWHID src = new SWHID(ctx.pathParam("src"));
- String dstFmt = ctx.pathParam("dst");
- String direction = ctx.queryParam("direction", "forward");
- String edgesFmt = ctx.queryParam("edges", "*");
- String algorithm = ctx.queryParam("traversal", "dfs");
-
- Endpoint endpoint = new Endpoint(graph, direction, edgesFmt);
- Endpoint.Output output = endpoint.walk(new Endpoint.Input(src, dstFmt, algorithm));
- ctx.json(formatEndpointOutput(output, showTimings));
- });
-
- app.exception(IllegalArgumentException.class, (e, ctx) -> {
- ctx.status(400);
- ctx.result(e.getMessage());
- });
- }
-
- /**
- * Checks query strings names provided to the RPC API.
- *
- * @param ctx Javalin HTTP request context
- * @param allowedFmt a regular expression describing allowed query strings names
- * @throws IllegalArgumentException unknown query string provided
- */
- private static void checkQueryStrings(Context ctx, String allowedFmt) {
- Map<String, List<String>> queryParamMap = ctx.queryParamMap();
- for (String key : queryParamMap.keySet()) {
- if (!key.matches(allowedFmt)) {
- throw new IllegalArgumentException("Unknown query string: " + key);
- }
- }
- }
-
- /**
- * Formats endpoint result into final JSON for the RPC API.
- * <p>
- * Removes unwanted information if necessary, such as timings (to prevent use of side channels
- * attacks).
- *
- * @param output endpoint operation output which needs formatting
- * @param showTimings true if timings should be in results metadata, false otherwise
- * @return final Object with desired JSON format
- */
- private static Object formatEndpointOutput(Endpoint.Output output, boolean showTimings) {
- if (showTimings) {
- return output;
- } else {
- Map<String, Object> metaNoTimings = Map.of("nb_edges_accessed", output.meta.nbEdgesAccessed);
- Map<String, Object> outputNoTimings = Map.of("result", output.result, "meta", metaNoTimings);
- return outputNoTimings;
- }
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/server/Endpoint.java b/java/src/main/java/org/softwareheritage/graph/server/Endpoint.java
deleted file mode 100644
--- a/java/src/main/java/org/softwareheritage/graph/server/Endpoint.java
+++ /dev/null
@@ -1,309 +0,0 @@
-package org.softwareheritage.graph.server;
-
-import org.softwareheritage.graph.*;
-import org.softwareheritage.graph.benchmark.utils.Timing;
-
-import java.util.ArrayList;
-
-/**
- * RPC API endpoints wrapper functions.
- * <p>
- * Graph operations are segmented between high-level class (this one) and the low-level class
- * ({@link Traversal}). The {@link Endpoint} class creates wrappers for each endpoints by performing
- * all the input/output node ids conversions and logging timings.
- *
- * @author The Software Heritage developers
- * @see Traversal
- */
-
-public class Endpoint {
- /** Graph where traversal endpoint is performed */
- SwhBidirectionalGraph graph;
- /** Internal traversal API */
- Traversal traversal;
-
- /**
- * Constructor.
- *
- * @param graph the graph used for traversal endpoint
- * @param direction a string (either "forward" or "backward") specifying edge orientation
- * @param edgesFmt a formatted string describing <a href=
- * "https://docs.softwareheritage.org/devel/swh-graph/api.html#terminology">allowed
- * edges</a>
- */
- public Endpoint(SwhBidirectionalGraph graph, String direction, String edgesFmt) {
- this.graph = graph;
- this.traversal = new Traversal(graph, direction, edgesFmt);
- }
-
- /**
- * Converts a list of (internal) long node ids to a list of corresponding (external) SWHIDs.
- *
- * @param nodeIds the list of long node ids
- * @return a list of corresponding SWHIDs
- */
- private ArrayList<SWHID> convertNodesToSWHIDs(ArrayList<Long> nodeIds) {
- ArrayList<SWHID> swhids = new ArrayList<>();
- for (long nodeId : nodeIds) {
- swhids.add(graph.getSWHID(nodeId));
- }
- return swhids;
- }
-
- /**
- * Converts a list of (internal) long node ids to the corresponding {@link SwhPath}.
- *
- * @param nodeIds the list of long node ids
- * @return the corresponding {@link SwhPath}
- * @see org.softwareheritage.graph.SwhPath
- */
- private SwhPath convertNodesToSwhPath(ArrayList<Long> nodeIds) {
- SwhPath path = new SwhPath();
- for (long nodeId : nodeIds) {
- path.add(graph.getSWHID(nodeId));
- }
- return path;
- }
-
- /**
- * Converts a list of paths made of (internal) long node ids to one made of {@link SwhPath}-s.
- *
- * @param pathsNodeId the list of paths with long node ids
- * @return a list of corresponding {@link SwhPath}
- * @see org.softwareheritage.graph.SwhPath
- */
- private ArrayList<SwhPath> convertPathsToSWHIDs(ArrayList<ArrayList<Long>> pathsNodeId) {
- ArrayList<SwhPath> paths = new ArrayList<>();
- for (ArrayList<Long> path : pathsNodeId) {
- paths.add(convertNodesToSwhPath(path));
- }
- return paths;
- }
-
- /**
- * Leaves endpoint wrapper.
- *
- * @param input input parameters for the underlying endpoint call
- * @return the resulting list of {@link SWHID} from endpoint call and operation metadata
- * @see SWHID
- * @see Traversal#leaves(long)
- */
- public Output leaves(Input input) {
- Output<ArrayList<SWHID>> output = new Output<>();
- long startTime;
-
- startTime = Timing.start();
- long srcNodeId = graph.getNodeId(input.src);
- output.meta.timings.swhid2node = Timing.stop(startTime);
-
- startTime = Timing.start();
- ArrayList<Long> nodeIds = traversal.leaves(srcNodeId);
- output.meta.timings.traversal = Timing.stop(startTime);
- output.meta.nbEdgesAccessed = traversal.getNbEdgesAccessed();
-
- startTime = Timing.start();
- output.result = convertNodesToSWHIDs(nodeIds);
- output.meta.timings.node2swhid = Timing.stop(startTime);
-
- return output;
- }
-
- /**
- * Neighbors endpoint wrapper.
- *
- * @param input input parameters for the underlying endpoint call
- * @return the resulting list of {@link SWHID} from endpoint call and operation metadata
- * @see SWHID
- * @see Traversal#neighbors(long)
- */
- public Output neighbors(Input input) {
- Output<ArrayList<SWHID>> output = new Output<>();
- long startTime;
-
- startTime = Timing.start();
- long srcNodeId = graph.getNodeId(input.src);
- output.meta.timings.swhid2node = Timing.stop(startTime);
-
- startTime = Timing.start();
- ArrayList<Long> nodeIds = traversal.neighbors(srcNodeId);
- output.meta.timings.traversal = Timing.stop(startTime);
- output.meta.nbEdgesAccessed = traversal.getNbEdgesAccessed();
-
- startTime = Timing.start();
- output.result = convertNodesToSWHIDs(nodeIds);
- output.meta.timings.node2swhid = Timing.stop(startTime);
-
- return output;
- }
-
- /**
- * Walk endpoint wrapper.
- *
- * @param input input parameters for the underlying endpoint call
- * @return the resulting {@link SwhPath} from endpoint call and operation metadata
- * @see SWHID
- * @see org.softwareheritage.graph.SwhPath
- * @see Traversal#walk
- */
- public Output walk(Input input) {
- Output<SwhPath> output = new Output<>();
- long startTime;
-
- startTime = Timing.start();
- long srcNodeId = graph.getNodeId(input.src);
- output.meta.timings.swhid2node = Timing.stop(startTime);
-
- ArrayList<Long> nodeIds = new ArrayList<Long>();
-
- // Destination is either a SWHID or a node type
- try {
- SWHID dstSWHID = new SWHID(input.dstFmt);
- long dstNodeId = graph.getNodeId(dstSWHID);
-
- startTime = Timing.start();
- nodeIds = traversal.walk(srcNodeId, dstNodeId, input.algorithm);
- output.meta.timings.traversal = Timing.stop(startTime);
- } catch (IllegalArgumentException ignored1) {
- try {
- Node.Type dstType = Node.Type.fromStr(input.dstFmt);
-
- startTime = Timing.start();
- nodeIds = traversal.walk(srcNodeId, dstType, input.algorithm);
- output.meta.timings.traversal = Timing.stop(startTime);
- } catch (IllegalArgumentException ignored2) {
- }
- }
-
- output.meta.nbEdgesAccessed = traversal.getNbEdgesAccessed();
-
- startTime = Timing.start();
- output.result = convertNodesToSwhPath(nodeIds);
- output.meta.timings.node2swhid = Timing.stop(startTime);
-
- return output;
- }
-
- /**
- * VisitNodes endpoint wrapper.
- *
- * @param input input parameters for the underlying endpoint call
- * @return the resulting list of {@link SWHID} from endpoint call and operation metadata
- * @see SWHID
- * @see Traversal#visitNodes(long)
- */
- public Output visitNodes(Input input) {
- Output<ArrayList<SWHID>> output = new Output<>();
- long startTime;
-
- startTime = Timing.start();
- long srcNodeId = graph.getNodeId(input.src);
- output.meta.timings.swhid2node = Timing.stop(startTime);
-
- startTime = Timing.start();
- ArrayList<Long> nodeIds = traversal.visitNodes(srcNodeId);
- output.meta.timings.traversal = Timing.stop(startTime);
- output.meta.nbEdgesAccessed = traversal.getNbEdgesAccessed();
-
- startTime = Timing.start();
- output.result = convertNodesToSWHIDs(nodeIds);
- output.meta.timings.node2swhid = Timing.stop(startTime);
-
- return output;
- }
-
- /**
- * VisitPaths endpoint wrapper.
- *
- * @param input input parameters for the underlying endpoint call
- * @return the resulting list of {@link SwhPath} from endpoint call and operation metadata
- * @see SWHID
- * @see org.softwareheritage.graph.SwhPath
- * @see Traversal#visitPaths(long)
- */
- public Output visitPaths(Input input) {
- Output<ArrayList<SwhPath>> output = new Output<>();
- long startTime;
-
- startTime = Timing.start();
- long srcNodeId = graph.getNodeId(input.src);
- output.meta.timings.swhid2node = Timing.stop(startTime);
-
- startTime = Timing.start();
- ArrayList<ArrayList<Long>> paths = traversal.visitPaths(srcNodeId);
- output.meta.timings.traversal = Timing.stop(startTime);
- output.meta.nbEdgesAccessed = traversal.getNbEdgesAccessed();
-
- startTime = Timing.start();
- output.result = convertPathsToSWHIDs(paths);
- output.meta.timings.node2swhid = Timing.stop(startTime);
-
- return output;
- }
-
- /**
- * Wrapper class to unify traversal methods input signatures.
- */
- public static class Input {
- /** Source node of endpoint call specified as a {@link SWHID} */
- public SWHID src;
- /**
- * Destination formatted string as described in the
- * <a href="https://docs.softwareheritage.org/devel/swh-graph/api.html#walk">API</a>
- */
- public String dstFmt;
- /** Traversal algorithm used in endpoint call (either "dfs" or "bfs") */
- public String algorithm;
-
- public Input(SWHID src) {
- this.src = src;
- }
-
- public Input(SWHID src, String dstFmt, String algorithm) {
- this.src = src;
- this.dstFmt = dstFmt;
- this.algorithm = algorithm;
- }
- }
-
- /**
- * Wrapper class to return both the endpoint result and metadata (such as timings).
- */
- public static class Output<T> {
- /** The result content itself */
- public T result;
- /** Various metadata about the result */
- public Meta meta;
-
- public Output() {
- this.result = null;
- this.meta = new Meta();
- }
-
- /**
- * Endpoint result metadata.
- */
- public class Meta {
- /** Operations timings */
- public Timings timings;
- /** Number of edges accessed during traversal */
- public long nbEdgesAccessed;
-
- public Meta() {
- this.timings = new Timings();
- this.nbEdgesAccessed = 0;
- }
-
- /**
- * Wrapper class for JSON output format.
- */
- public class Timings {
- /** Time in seconds to do the traversal */
- public double traversal;
- /** Time in seconds to convert input SWHID to node id */
- public double swhid2node;
- /** Time in seconds to convert output node ids to SWHIDs */
- public double node2swhid;
- }
- }
- }
-}
diff --git a/java/src/main/java/org/softwareheritage/graph/utils/FindEarliestRevision.java b/java/src/main/java/org/softwareheritage/graph/utils/FindEarliestRevision.java
--- a/java/src/main/java/org/softwareheritage/graph/utils/FindEarliestRevision.java
+++ b/java/src/main/java/org/softwareheritage/graph/utils/FindEarliestRevision.java
@@ -84,8 +84,11 @@
}
}
- LazyLongIterator it = Traversal.filterSuccessors(graph, currentNodeId, edges);
+ LazyLongIterator it = graph.successors(currentNodeId);
for (long neighborNodeId; (neighborNodeId = it.nextLong()) != -1;) {
+ if (!edges.isAllowed(graph.getNodeType(currentNodeId), graph.getNodeType(neighborNodeId))) {
+ continue;
+ }
if (!visited.contains(neighborNodeId)) {
stack.push(neighborNodeId);
visited.add(neighborNodeId);
diff --git a/java/src/main/proto b/java/src/main/proto
new file mode 120000
--- /dev/null
+++ b/java/src/main/proto
@@ -0,0 +1 @@
+../../../proto
\ No newline at end of file
diff --git a/java/src/test/java/org/softwareheritage/graph/GraphTest.java b/java/src/test/java/org/softwareheritage/graph/GraphTest.java
--- a/java/src/test/java/org/softwareheritage/graph/GraphTest.java
+++ b/java/src/test/java/org/softwareheritage/graph/GraphTest.java
@@ -6,15 +6,15 @@
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Collection;
+import java.util.Comparator;
import java.util.Iterator;
import com.github.luben.zstd.ZstdInputStream;
import it.unimi.dsi.big.webgraph.LazyLongIterator;
import it.unimi.dsi.big.webgraph.LazyLongIterators;
-import org.hamcrest.MatcherAssert;
import org.junit.jupiter.api.BeforeAll;
-import static org.hamcrest.collection.IsIterableContainingInAnyOrder.containsInAnyOrder;
+import static org.junit.Assert.assertEquals;
public class GraphTest {
static SwhBidirectionalGraph graph;
@@ -23,11 +23,14 @@
@BeforeAll
public static void setUp() throws IOException {
- Path graphPath = Paths.get("..", "swh", "graph", "tests", "dataset", "compressed", "example");
- graph = SwhBidirectionalGraph.loadMapped(graphPath.toString());
+ graph = SwhBidirectionalGraph.loadLabelled(getGraphPath().toString());
}
- public SwhBidirectionalGraph getGraph() {
+ public static Path getGraphPath() {
+ return Paths.get("..", "swh", "graph", "tests", "dataset", "compressed", "example");
+ }
+
+ public static SwhBidirectionalGraph getGraph() {
return graph;
}
@@ -35,8 +38,12 @@
return new SWHID(String.format("swh:1:%s:%040d", type, num));
}
- public static <T> void assertEqualsAnyOrder(Collection<T> expecteds, Collection<T> actuals) {
- MatcherAssert.assertThat(expecteds, containsInAnyOrder(actuals.toArray()));
+ public static <T> void assertEqualsAnyOrder(Collection<T> expected, Collection<T> actual) {
+ ArrayList<T> expectedList = new ArrayList<>(expected);
+ ArrayList<T> actualList = new ArrayList<>(actual);
+ expectedList.sort(Comparator.comparing(Object::toString));
+ actualList.sort(Comparator.comparing(Object::toString));
+ assertEquals(expectedList, actualList);
}
public static ArrayList<Long> lazyLongIteratorToList(LazyLongIterator input) {
diff --git a/java/src/test/java/org/softwareheritage/graph/NeighborsTest.java b/java/src/test/java/org/softwareheritage/graph/NeighborsTest.java
deleted file mode 100644
--- a/java/src/test/java/org/softwareheritage/graph/NeighborsTest.java
+++ /dev/null
@@ -1,141 +0,0 @@
-package org.softwareheritage.graph;
-
-import java.util.ArrayList;
-
-import org.junit.jupiter.api.Test;
-import org.softwareheritage.graph.server.Endpoint;
-
-// Avoid warnings concerning Endpoint.Output.result manual cast
-@SuppressWarnings("unchecked")
-public class NeighborsTest extends GraphTest {
- @Test
- public void zeroNeighbor() {
- SwhBidirectionalGraph graph = getGraph();
- ArrayList<SWHID> expectedNodes = new ArrayList<>();
-
- SWHID src1 = new SWHID(TEST_ORIGIN_ID);
- Endpoint endpoint1 = new Endpoint(graph, "backward", "*");
- ArrayList<SWHID> actuals1 = (ArrayList) endpoint1.neighbors(new Endpoint.Input(src1)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes, actuals1);
-
- SWHID src2 = new SWHID("swh:1:cnt:0000000000000000000000000000000000000004");
- Endpoint endpoint2 = new Endpoint(graph, "forward", "*");
- ArrayList<SWHID> actuals2 = (ArrayList) endpoint2.neighbors(new Endpoint.Input(src2)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes, actuals2);
-
- SWHID src3 = new SWHID("swh:1:cnt:0000000000000000000000000000000000000015");
- Endpoint endpoint3 = new Endpoint(graph, "forward", "*");
- ArrayList<SWHID> actuals3 = (ArrayList) endpoint3.neighbors(new Endpoint.Input(src3)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes, actuals3);
-
- SWHID src4 = new SWHID("swh:1:rel:0000000000000000000000000000000000000019");
- Endpoint endpoint4 = new Endpoint(graph, "backward", "*");
- ArrayList<SWHID> actuals4 = (ArrayList) endpoint4.neighbors(new Endpoint.Input(src4)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes, actuals4);
-
- SWHID src5 = new SWHID("swh:1:dir:0000000000000000000000000000000000000008");
- Endpoint endpoint5 = new Endpoint(graph, "forward", "snp:*,rev:*,rel:*");
- ArrayList<SWHID> actuals5 = (ArrayList) endpoint5.neighbors(new Endpoint.Input(src5)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes, actuals5);
- }
-
- @Test
- public void oneNeighbor() {
- SwhBidirectionalGraph graph = getGraph();
-
- SWHID src1 = new SWHID("swh:1:rev:0000000000000000000000000000000000000003");
- Endpoint endpoint1 = new Endpoint(graph, "forward", "*");
- ArrayList<SWHID> expectedNodes1 = new ArrayList<>();
- expectedNodes1.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000002"));
- ArrayList<SWHID> actuals1 = (ArrayList) endpoint1.neighbors(new Endpoint.Input(src1)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes1, actuals1);
-
- SWHID src2 = new SWHID("swh:1:dir:0000000000000000000000000000000000000017");
- Endpoint endpoint2 = new Endpoint(graph, "forward", "dir:cnt");
- ArrayList<SWHID> expectedNodes2 = new ArrayList<>();
- expectedNodes2.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000014"));
- ArrayList<SWHID> actuals2 = (ArrayList) endpoint2.neighbors(new Endpoint.Input(src2)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes2, actuals2);
-
- SWHID src3 = new SWHID("swh:1:dir:0000000000000000000000000000000000000012");
- Endpoint endpoint3 = new Endpoint(graph, "backward", "*");
- ArrayList<SWHID> expectedNodes3 = new ArrayList<>();
- expectedNodes3.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000013"));
- ArrayList<SWHID> actuals3 = (ArrayList) endpoint3.neighbors(new Endpoint.Input(src3)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes3, actuals3);
-
- SWHID src4 = new SWHID("swh:1:rev:0000000000000000000000000000000000000009");
- Endpoint endpoint4 = new Endpoint(graph, "backward", "rev:rev");
- ArrayList<SWHID> expectedNodes4 = new ArrayList<>();
- expectedNodes4.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000013"));
- ArrayList<SWHID> actuals4 = (ArrayList) endpoint4.neighbors(new Endpoint.Input(src4)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes4, actuals4);
-
- SWHID src5 = new SWHID("swh:1:snp:0000000000000000000000000000000000000020");
- Endpoint endpoint5 = new Endpoint(graph, "backward", "*");
- ArrayList<SWHID> expectedNodes5 = new ArrayList<>();
- expectedNodes5.add(new SWHID(TEST_ORIGIN_ID));
- ArrayList<SWHID> actuals5 = (ArrayList) endpoint5.neighbors(new Endpoint.Input(src5)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes5, actuals5);
- }
-
- @Test
- public void twoNeighbors() {
- SwhBidirectionalGraph graph = getGraph();
-
- SWHID src1 = new SWHID("swh:1:snp:0000000000000000000000000000000000000020");
- Endpoint endpoint1 = new Endpoint(graph, "forward", "*");
- ArrayList<SWHID> expectedNodes1 = new ArrayList<>();
- expectedNodes1.add(new SWHID("swh:1:rel:0000000000000000000000000000000000000010"));
- expectedNodes1.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000009"));
- ArrayList<SWHID> actuals1 = (ArrayList) endpoint1.neighbors(new Endpoint.Input(src1)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes1, actuals1);
-
- SWHID src2 = new SWHID("swh:1:dir:0000000000000000000000000000000000000008");
- Endpoint endpoint2 = new Endpoint(graph, "forward", "dir:cnt");
- ArrayList<SWHID> expectedNodes2 = new ArrayList<>();
- expectedNodes2.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000001"));
- expectedNodes2.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000007"));
- ArrayList<SWHID> actuals2 = (ArrayList) endpoint2.neighbors(new Endpoint.Input(src2)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes2, actuals2);
-
- SWHID src3 = new SWHID("swh:1:cnt:0000000000000000000000000000000000000001");
- Endpoint endpoint3 = new Endpoint(graph, "backward", "*");
- ArrayList<SWHID> expectedNodes3 = new ArrayList<>();
- expectedNodes3.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000008"));
- expectedNodes3.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000002"));
- ArrayList<SWHID> actuals3 = (ArrayList) endpoint3.neighbors(new Endpoint.Input(src3)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes3, actuals3);
-
- SWHID src4 = new SWHID("swh:1:rev:0000000000000000000000000000000000000009");
- Endpoint endpoint4 = new Endpoint(graph, "backward", "rev:snp,rev:rel");
- ArrayList<SWHID> expectedNodes4 = new ArrayList<>();
- expectedNodes4.add(new SWHID("swh:1:snp:0000000000000000000000000000000000000020"));
- expectedNodes4.add(new SWHID("swh:1:rel:0000000000000000000000000000000000000010"));
- ArrayList<SWHID> actuals4 = (ArrayList) endpoint4.neighbors(new Endpoint.Input(src4)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes4, actuals4);
- }
-
- @Test
- public void threeNeighbors() {
- SwhBidirectionalGraph graph = getGraph();
-
- SWHID src1 = new SWHID("swh:1:dir:0000000000000000000000000000000000000008");
- Endpoint endpoint1 = new Endpoint(graph, "forward", "*");
- ArrayList<SWHID> expectedNodes1 = new ArrayList<>();
- expectedNodes1.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000006"));
- expectedNodes1.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000001"));
- expectedNodes1.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000007"));
- ArrayList<SWHID> actuals1 = (ArrayList) endpoint1.neighbors(new Endpoint.Input(src1)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes1, actuals1);
-
- SWHID src2 = new SWHID("swh:1:rev:0000000000000000000000000000000000000009");
- Endpoint endpoint2 = new Endpoint(graph, "backward", "*");
- ArrayList<SWHID> expectedNodes2 = new ArrayList<>();
- expectedNodes2.add(new SWHID("swh:1:snp:0000000000000000000000000000000000000020"));
- expectedNodes2.add(new SWHID("swh:1:rel:0000000000000000000000000000000000000010"));
- expectedNodes2.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000013"));
- ArrayList<SWHID> actuals2 = (ArrayList) endpoint2.neighbors(new Endpoint.Input(src2)).result;
- GraphTest.assertEqualsAnyOrder(expectedNodes2, actuals2);
- }
-}
diff --git a/java/src/test/java/org/softwareheritage/graph/VisitTest.java b/java/src/test/java/org/softwareheritage/graph/VisitTest.java
deleted file mode 100644
--- a/java/src/test/java/org/softwareheritage/graph/VisitTest.java
+++ /dev/null
@@ -1,408 +0,0 @@
-package org.softwareheritage.graph;
-
-import java.util.ArrayList;
-import java.util.Set;
-import java.util.HashSet;
-
-import org.junit.jupiter.api.Test;
-import org.softwareheritage.graph.server.Endpoint;
-
-// Avoid warnings concerning Endpoint.Output.result manual cast
-@SuppressWarnings("unchecked")
-public class VisitTest extends GraphTest {
- private void assertSameNodesFromPaths(ArrayList<SwhPath> paths, ArrayList<SWHID> nodes) {
- Set<SWHID> expectedNodes = new HashSet<SWHID>();
- for (SwhPath path : paths) {
- expectedNodes.addAll(path.getPath());
- }
- GraphTest.assertEqualsAnyOrder(expectedNodes, nodes);
- }
-
- @Test
- public void forwardFromRoot() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID(TEST_ORIGIN_ID);
- Endpoint endpoint1 = new Endpoint(graph, "forward", "*");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "*");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath(TEST_ORIGIN_ID, "swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000007"));
- expectedPaths.add(new SwhPath(TEST_ORIGIN_ID, "swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000001"));
- expectedPaths.add(new SwhPath(TEST_ORIGIN_ID, "swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000004"));
- expectedPaths.add(new SwhPath(TEST_ORIGIN_ID, "swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000005"));
- expectedPaths.add(new SwhPath(TEST_ORIGIN_ID, "swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rev:0000000000000000000000000000000000000003",
- "swh:1:dir:0000000000000000000000000000000000000002",
- "swh:1:cnt:0000000000000000000000000000000000000001"));
- expectedPaths.add(new SwhPath(TEST_ORIGIN_ID, "swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rel:0000000000000000000000000000000000000010",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000007"));
- expectedPaths.add(new SwhPath(TEST_ORIGIN_ID, "swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rel:0000000000000000000000000000000000000010",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000001"));
- expectedPaths.add(new SwhPath(TEST_ORIGIN_ID, "swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rel:0000000000000000000000000000000000000010",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000004"));
- expectedPaths.add(new SwhPath(TEST_ORIGIN_ID, "swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rel:0000000000000000000000000000000000000010",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000005"));
- expectedPaths.add(new SwhPath(TEST_ORIGIN_ID, "swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rel:0000000000000000000000000000000000000010",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rev:0000000000000000000000000000000000000003",
- "swh:1:dir:0000000000000000000000000000000000000002",
- "swh:1:cnt:0000000000000000000000000000000000000001"));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void forwardFromMiddle() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID("swh:1:dir:0000000000000000000000000000000000000012");
- Endpoint endpoint1 = new Endpoint(graph, "forward", "*");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "*");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath("swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000007"));
- expectedPaths.add(new SwhPath("swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000001"));
- expectedPaths.add(new SwhPath("swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000004"));
- expectedPaths.add(new SwhPath("swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000005"));
- expectedPaths.add(new SwhPath("swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:cnt:0000000000000000000000000000000000000011"));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void forwardFromLeaf() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID("swh:1:cnt:0000000000000000000000000000000000000004");
- Endpoint endpoint1 = new Endpoint(graph, "forward", "*");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "*");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath("swh:1:cnt:0000000000000000000000000000000000000004"));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void backwardFromRoot() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID(TEST_ORIGIN_ID);
- Endpoint endpoint1 = new Endpoint(graph, "backward", "*");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "backward", "*");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath(TEST_ORIGIN_ID));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void backwardFromMiddle() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID("swh:1:dir:0000000000000000000000000000000000000012");
- Endpoint endpoint1 = new Endpoint(graph, "backward", "*");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "backward", "*");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath("swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:rev:0000000000000000000000000000000000000018",
- "swh:1:rel:0000000000000000000000000000000000000019"));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void backwardFromLeaf() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID("swh:1:cnt:0000000000000000000000000000000000000004");
- Endpoint endpoint1 = new Endpoint(graph, "backward", "*");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "backward", "*");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath("swh:1:cnt:0000000000000000000000000000000000000004",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:rev:0000000000000000000000000000000000000018",
- "swh:1:rel:0000000000000000000000000000000000000019"));
- expectedPaths.add(new SwhPath("swh:1:cnt:0000000000000000000000000000000000000004",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:rev:0000000000000000000000000000000000000018",
- "swh:1:rel:0000000000000000000000000000000000000019"));
- expectedPaths.add(new SwhPath("swh:1:cnt:0000000000000000000000000000000000000004",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:snp:0000000000000000000000000000000000000020", TEST_ORIGIN_ID));
- expectedPaths.add(new SwhPath("swh:1:cnt:0000000000000000000000000000000000000004",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rel:0000000000000000000000000000000000000010",
- "swh:1:snp:0000000000000000000000000000000000000020", TEST_ORIGIN_ID));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void forwardSnpToRev() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID("swh:1:snp:0000000000000000000000000000000000000020");
- Endpoint endpoint1 = new Endpoint(graph, "forward", "snp:rev");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "snp:rev");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath("swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rev:0000000000000000000000000000000000000009"));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void forwardRelToRevRevToRev() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID("swh:1:rel:0000000000000000000000000000000000000010");
- Endpoint endpoint1 = new Endpoint(graph, "forward", "rel:rev,rev:rev");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "rel:rev,rev:rev");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath("swh:1:rel:0000000000000000000000000000000000000010",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rev:0000000000000000000000000000000000000003"));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void forwardRevToAllDirToAll() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID("swh:1:rev:0000000000000000000000000000000000000013");
- Endpoint endpoint1 = new Endpoint(graph, "forward", "rev:*,dir:*");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "rev:*,dir:*");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000005"));
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000005"));
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000004"));
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000004"));
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000007"));
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000007"));
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:cnt:0000000000000000000000000000000000000011"));
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rev:0000000000000000000000000000000000000003",
- "swh:1:dir:0000000000000000000000000000000000000002",
- "swh:1:cnt:0000000000000000000000000000000000000001"));
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000001"));
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:dir:0000000000000000000000000000000000000012",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000001"));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void forwardSnpToAllRevToAll() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID("swh:1:snp:0000000000000000000000000000000000000020");
- Endpoint endpoint1 = new Endpoint(graph, "forward", "snp:*,rev:*");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "snp:*,rev:*");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath("swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rev:0000000000000000000000000000000000000003",
- "swh:1:dir:0000000000000000000000000000000000000002"));
- expectedPaths.add(new SwhPath("swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008"));
- expectedPaths.add(new SwhPath("swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rel:0000000000000000000000000000000000000010"));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void forwardNoEdges() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID("swh:1:snp:0000000000000000000000000000000000000020");
- Endpoint endpoint1 = new Endpoint(graph, "forward", "");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath("swh:1:snp:0000000000000000000000000000000000000020"));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void backwardRevToRevRevToRel() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID("swh:1:rev:0000000000000000000000000000000000000003");
- Endpoint endpoint1 = new Endpoint(graph, "backward", "rev:rev,rev:rel");
- ArrayList<SwhPath> paths = (ArrayList) endpoint1.visitPaths(new Endpoint.Input(swhid)).result;
- Endpoint endpoint2 = new Endpoint(graph, "backward", "rev:rev,rev:rel");
- ArrayList<SWHID> nodes = (ArrayList) endpoint2.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SwhPath> expectedPaths = new ArrayList<SwhPath>();
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000003",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:rev:0000000000000000000000000000000000000018",
- "swh:1:rel:0000000000000000000000000000000000000019"));
- expectedPaths.add(new SwhPath("swh:1:rev:0000000000000000000000000000000000000003",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rel:0000000000000000000000000000000000000010"));
-
- GraphTest.assertEqualsAnyOrder(expectedPaths, paths);
- assertSameNodesFromPaths(expectedPaths, nodes);
- }
-
- @Test
- public void forwardFromRootNodesOnly() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID(TEST_ORIGIN_ID);
- Endpoint endpoint = new Endpoint(graph, "forward", "*");
- ArrayList<SWHID> nodes = (ArrayList) endpoint.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SWHID> expectedNodes = new ArrayList<SWHID>();
- expectedNodes.add(new SWHID(TEST_ORIGIN_ID));
- expectedNodes.add(new SWHID("swh:1:snp:0000000000000000000000000000000000000020"));
- expectedNodes.add(new SWHID("swh:1:rel:0000000000000000000000000000000000000010"));
- expectedNodes.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000009"));
- expectedNodes.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000003"));
- expectedNodes.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000002"));
- expectedNodes.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000001"));
- expectedNodes.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000008"));
- expectedNodes.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000006"));
- expectedNodes.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000004"));
- expectedNodes.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000005"));
- expectedNodes.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000007"));
-
- GraphTest.assertEqualsAnyOrder(expectedNodes, nodes);
- }
-
- @Test
- public void backwardRevToAllNodesOnly() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID swhid = new SWHID("swh:1:rev:0000000000000000000000000000000000000003");
- Endpoint endpoint = new Endpoint(graph, "backward", "rev:*");
- ArrayList<SWHID> nodes = (ArrayList) endpoint.visitNodes(new Endpoint.Input(swhid)).result;
-
- ArrayList<SWHID> expectedNodes = new ArrayList<SWHID>();
- expectedNodes.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000003"));
- expectedNodes.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000009"));
- expectedNodes.add(new SWHID("swh:1:snp:0000000000000000000000000000000000000020"));
- expectedNodes.add(new SWHID("swh:1:rel:0000000000000000000000000000000000000010"));
- expectedNodes.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000013"));
- expectedNodes.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000018"));
- expectedNodes.add(new SWHID("swh:1:rel:0000000000000000000000000000000000000019"));
-
- GraphTest.assertEqualsAnyOrder(expectedNodes, nodes);
- }
-}
diff --git a/java/src/test/java/org/softwareheritage/graph/WalkTest.java b/java/src/test/java/org/softwareheritage/graph/WalkTest.java
deleted file mode 100644
--- a/java/src/test/java/org/softwareheritage/graph/WalkTest.java
+++ /dev/null
@@ -1,187 +0,0 @@
-package org.softwareheritage.graph;
-
-import java.util.Arrays;
-import java.util.List;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-import org.softwareheritage.graph.server.Endpoint;
-
-public class WalkTest extends GraphTest {
- @Test
- public void forwardRootToLeaf() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:snp:0000000000000000000000000000000000000020");
- String dstFmt = "swh:1:cnt:0000000000000000000000000000000000000005";
-
- SwhPath solution1 = new SwhPath("swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000005");
- SwhPath solution2 = new SwhPath("swh:1:snp:0000000000000000000000000000000000000020",
- "swh:1:rel:0000000000000000000000000000000000000010",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000005");
-
- Endpoint endpoint1 = new Endpoint(graph, "forward", "*");
- SwhPath dfsPath = (SwhPath) endpoint1.walk(new Endpoint.Input(src, dstFmt, "dfs")).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "*");
- SwhPath bfsPath = (SwhPath) endpoint2.walk(new Endpoint.Input(src, dstFmt, "bfs")).result;
-
- List<SwhPath> possibleSolutions = Arrays.asList(solution1, solution2);
- Assertions.assertTrue(possibleSolutions.contains(dfsPath));
- Assertions.assertTrue(possibleSolutions.contains(bfsPath));
- }
-
- @Test
- public void forwardLeafToLeaf() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:cnt:0000000000000000000000000000000000000007");
- String dstFmt = "cnt";
-
- SwhPath expectedPath = new SwhPath("swh:1:cnt:0000000000000000000000000000000000000007");
-
- Endpoint endpoint1 = new Endpoint(graph, "forward", "*");
- SwhPath dfsPath = (SwhPath) endpoint1.walk(new Endpoint.Input(src, dstFmt, "dfs")).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "*");
- SwhPath bfsPath = (SwhPath) endpoint2.walk(new Endpoint.Input(src, dstFmt, "bfs")).result;
-
- Assertions.assertEquals(dfsPath, expectedPath);
- Assertions.assertEquals(bfsPath, expectedPath);
- }
-
- @Test
- public void forwardRevToRev() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:rev:0000000000000000000000000000000000000018");
- String dstFmt = "swh:1:rev:0000000000000000000000000000000000000003";
-
- SwhPath expectedPath = new SwhPath("swh:1:rev:0000000000000000000000000000000000000018",
- "swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rev:0000000000000000000000000000000000000003");
-
- Endpoint endpoint1 = new Endpoint(graph, "forward", "rev:rev");
- SwhPath dfsPath = (SwhPath) endpoint1.walk(new Endpoint.Input(src, dstFmt, "dfs")).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "rev:rev");
- SwhPath bfsPath = (SwhPath) endpoint2.walk(new Endpoint.Input(src, dstFmt, "bfs")).result;
-
- Assertions.assertEquals(dfsPath, expectedPath);
- Assertions.assertEquals(bfsPath, expectedPath);
- }
-
- @Test
- public void backwardRevToRev() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:rev:0000000000000000000000000000000000000003");
- String dstFmt = "swh:1:rev:0000000000000000000000000000000000000018";
-
- SwhPath expectedPath = new SwhPath("swh:1:rev:0000000000000000000000000000000000000003",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rev:0000000000000000000000000000000000000013",
- "swh:1:rev:0000000000000000000000000000000000000018");
-
- Endpoint endpoint1 = new Endpoint(graph, "backward", "rev:rev");
- SwhPath dfsPath = (SwhPath) endpoint1.walk(new Endpoint.Input(src, dstFmt, "dfs")).result;
- Endpoint endpoint2 = new Endpoint(graph, "backward", "rev:rev");
- SwhPath bfsPath = (SwhPath) endpoint2.walk(new Endpoint.Input(src, dstFmt, "bfs")).result;
-
- Assertions.assertEquals(dfsPath, expectedPath);
- Assertions.assertEquals(bfsPath, expectedPath);
- }
-
- @Test
- public void backwardCntToFirstSnp() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:cnt:0000000000000000000000000000000000000001");
- String dstFmt = "snp";
-
- SwhPath solution1 = new SwhPath("swh:1:cnt:0000000000000000000000000000000000000001",
- "swh:1:dir:0000000000000000000000000000000000000002",
- "swh:1:rev:0000000000000000000000000000000000000003",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:snp:0000000000000000000000000000000000000020");
- SwhPath solution2 = new SwhPath("swh:1:cnt:0000000000000000000000000000000000000001",
- "swh:1:dir:0000000000000000000000000000000000000002",
- "swh:1:rev:0000000000000000000000000000000000000003",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rel:0000000000000000000000000000000000000010",
- "swh:1:snp:0000000000000000000000000000000000000020");
- SwhPath solution3 = new SwhPath("swh:1:cnt:0000000000000000000000000000000000000001",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:snp:0000000000000000000000000000000000000020");
- SwhPath solution4 = new SwhPath("swh:1:cnt:0000000000000000000000000000000000000001",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rel:0000000000000000000000000000000000000010",
- "swh:1:snp:0000000000000000000000000000000000000020");
-
- Endpoint endpoint1 = new Endpoint(graph, "backward", "*");
- SwhPath dfsPath = (SwhPath) endpoint1.walk(new Endpoint.Input(src, dstFmt, "dfs")).result;
- Endpoint endpoint2 = new Endpoint(graph, "backward", "*");
- SwhPath bfsPath = (SwhPath) endpoint2.walk(new Endpoint.Input(src, dstFmt, "bfs")).result;
-
- List<SwhPath> possibleSolutions = Arrays.asList(solution1, solution2, solution3, solution4);
- Assertions.assertTrue(possibleSolutions.contains(dfsPath));
- Assertions.assertTrue(possibleSolutions.contains(bfsPath));
- }
-
- @Test
- public void forwardRevToFirstCnt() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:rev:0000000000000000000000000000000000000009");
- String dstFmt = "cnt";
-
- SwhPath solution1 = new SwhPath("swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000007");
- SwhPath solution2 = new SwhPath("swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000005");
- SwhPath solution3 = new SwhPath("swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:dir:0000000000000000000000000000000000000006",
- "swh:1:cnt:0000000000000000000000000000000000000004");
- SwhPath solution4 = new SwhPath("swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:dir:0000000000000000000000000000000000000008",
- "swh:1:cnt:0000000000000000000000000000000000000001");
- SwhPath solution5 = new SwhPath("swh:1:rev:0000000000000000000000000000000000000009",
- "swh:1:rev:0000000000000000000000000000000000000003",
- "swh:1:dir:0000000000000000000000000000000000000002",
- "swh:1:cnt:0000000000000000000000000000000000000001");
-
- Endpoint endpoint1 = new Endpoint(graph, "forward", "rev:*,dir:*");
- SwhPath dfsPath = (SwhPath) endpoint1.walk(new Endpoint.Input(src, dstFmt, "dfs")).result;
- Endpoint endpoint2 = new Endpoint(graph, "forward", "rev:*,dir:*");
- SwhPath bfsPath = (SwhPath) endpoint2.walk(new Endpoint.Input(src, dstFmt, "bfs")).result;
-
- List<SwhPath> possibleSolutions = Arrays.asList(solution1, solution2, solution3, solution4, solution5);
- Assertions.assertTrue(possibleSolutions.contains(dfsPath));
- Assertions.assertTrue(possibleSolutions.contains(bfsPath));
- }
-
- @Test
- public void backwardDirToFirstRel() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:dir:0000000000000000000000000000000000000016");
- String dstFmt = "rel";
-
- SwhPath expectedPath = new SwhPath("swh:1:dir:0000000000000000000000000000000000000016",
- "swh:1:dir:0000000000000000000000000000000000000017",
- "swh:1:rev:0000000000000000000000000000000000000018",
- "swh:1:rel:0000000000000000000000000000000000000019");
-
- Endpoint endpoint1 = new Endpoint(graph, "backward", "dir:dir,dir:rev,rev:*");
- SwhPath dfsPath = (SwhPath) endpoint1.walk(new Endpoint.Input(src, dstFmt, "dfs")).result;
- Endpoint endpoint2 = new Endpoint(graph, "backward", "dir:dir,dir:rev,rev:*");
- SwhPath bfsPath = (SwhPath) endpoint2.walk(new Endpoint.Input(src, dstFmt, "bfs")).result;
-
- Assertions.assertEquals(dfsPath, expectedPath);
- Assertions.assertEquals(bfsPath, expectedPath);
- }
-}
diff --git a/java/src/test/java/org/softwareheritage/graph/rpc/CountEdgesTest.java b/java/src/test/java/org/softwareheritage/graph/rpc/CountEdgesTest.java
new file mode 100644
--- /dev/null
+++ b/java/src/test/java/org/softwareheritage/graph/rpc/CountEdgesTest.java
@@ -0,0 +1,84 @@
+/*
+ * Copyright (c) 2022 The Software Heritage developers
+ * See the AUTHORS file at the top-level directory of this distribution
+ * License: GNU General Public License version 3, or any later version
+ * See top-level LICENSE file for more information
+ */
+
+package org.softwareheritage.graph.rpc;
+
+import com.google.protobuf.FieldMask;
+import io.grpc.Status;
+import io.grpc.StatusRuntimeException;
+import org.junit.jupiter.api.Test;
+import org.softwareheritage.graph.SWHID;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class CountEdgesTest extends TraversalServiceTest {
+ private TraversalRequest.Builder getTraversalRequestBuilder(SWHID src) {
+ return TraversalRequest.newBuilder().addSrc(src.toString());
+ }
+
+ @Test
+ public void testSwhidErrors() {
+ StatusRuntimeException thrown;
+ thrown = assertThrows(StatusRuntimeException.class, () -> client
+ .countEdges(TraversalRequest.newBuilder().addSrc(fakeSWHID("cnt", 404).toString()).build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.countEdges(
+ TraversalRequest.newBuilder().addSrc("swh:1:lol:0000000000000000000000000000000000000001").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.countEdges(
+ TraversalRequest.newBuilder().addSrc("swh:1:cnt:000000000000000000000000000000000000000z").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ }
+
+ @Test
+ public void forwardFromRoot() {
+ CountResponse actual = client.countEdges(getTraversalRequestBuilder(new SWHID(TEST_ORIGIN_ID)).build());
+ assertEquals(13, actual.getCount());
+ }
+
+ @Test
+ public void forwardFromMiddle() {
+ CountResponse actual = client.countEdges(getTraversalRequestBuilder(fakeSWHID("dir", 12)).build());
+ assertEquals(7, actual.getCount());
+ }
+
+ @Test
+ public void forwardRelRev() {
+ CountResponse actual = client
+ .countEdges(getTraversalRequestBuilder(fakeSWHID("rel", 10)).setEdges("rel:rev,rev:rev").build());
+ assertEquals(2, actual.getCount());
+ }
+
+ @Test
+ public void backwardFromMiddle() {
+ CountResponse actual = client.countEdges(
+ getTraversalRequestBuilder(fakeSWHID("dir", 12)).setDirection(GraphDirection.BACKWARD).build());
+ assertEquals(3, actual.getCount());
+ }
+
+ @Test
+ public void backwardFromLeaf() {
+ CountResponse actual = client.countEdges(
+ getTraversalRequestBuilder(fakeSWHID("cnt", 4)).setDirection(GraphDirection.BACKWARD).build());
+ assertEquals(12, actual.getCount());
+ }
+
+ @Test
+ public void backwardRevToRevRevToRel() {
+ CountResponse actual = client.countEdges(getTraversalRequestBuilder(fakeSWHID("rev", 3))
+ .setEdges("rev:rev,rev:rel").setDirection(GraphDirection.BACKWARD).build());
+ assertEquals(5, actual.getCount());
+ }
+
+ @Test
+ public void testWithEmptyMask() {
+ CountResponse actual = client.countEdges(
+ getTraversalRequestBuilder(fakeSWHID("dir", 12)).setMask(FieldMask.getDefaultInstance()).build());
+ assertEquals(7, actual.getCount());
+ }
+}
diff --git a/java/src/test/java/org/softwareheritage/graph/rpc/CountNodesTest.java b/java/src/test/java/org/softwareheritage/graph/rpc/CountNodesTest.java
new file mode 100644
--- /dev/null
+++ b/java/src/test/java/org/softwareheritage/graph/rpc/CountNodesTest.java
@@ -0,0 +1,84 @@
+/*
+ * Copyright (c) 2022 The Software Heritage developers
+ * See the AUTHORS file at the top-level directory of this distribution
+ * License: GNU General Public License version 3, or any later version
+ * See top-level LICENSE file for more information
+ */
+
+package org.softwareheritage.graph.rpc;
+
+import com.google.protobuf.FieldMask;
+import io.grpc.Status;
+import io.grpc.StatusRuntimeException;
+import org.junit.jupiter.api.Test;
+import org.softwareheritage.graph.SWHID;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class CountNodesTest extends TraversalServiceTest {
+ private TraversalRequest.Builder getTraversalRequestBuilder(SWHID src) {
+ return TraversalRequest.newBuilder().addSrc(src.toString());
+ }
+
+ @Test
+ public void testSwhidErrors() {
+ StatusRuntimeException thrown;
+ thrown = assertThrows(StatusRuntimeException.class, () -> client
+ .countNodes(TraversalRequest.newBuilder().addSrc(fakeSWHID("cnt", 404).toString()).build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.countNodes(
+ TraversalRequest.newBuilder().addSrc("swh:1:lol:0000000000000000000000000000000000000001").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.countNodes(
+ TraversalRequest.newBuilder().addSrc("swh:1:cnt:000000000000000000000000000000000000000z").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ }
+
+ @Test
+ public void forwardFromRoot() {
+ CountResponse actual = client.countNodes(getTraversalRequestBuilder(new SWHID(TEST_ORIGIN_ID)).build());
+ assertEquals(12, actual.getCount());
+ }
+
+ @Test
+ public void forwardFromMiddle() {
+ CountResponse actual = client.countNodes(getTraversalRequestBuilder(fakeSWHID("dir", 12)).build());
+ assertEquals(8, actual.getCount());
+ }
+
+ @Test
+ public void forwardRelRev() {
+ CountResponse actual = client
+ .countNodes(getTraversalRequestBuilder(fakeSWHID("rel", 10)).setEdges("rel:rev,rev:rev").build());
+ assertEquals(3, actual.getCount());
+ }
+
+ @Test
+ public void backwardFromMiddle() {
+ CountResponse actual = client.countNodes(
+ getTraversalRequestBuilder(fakeSWHID("dir", 12)).setDirection(GraphDirection.BACKWARD).build());
+ assertEquals(4, actual.getCount());
+ }
+
+ @Test
+ public void backwardFromLeaf() {
+ CountResponse actual = client.countNodes(
+ getTraversalRequestBuilder(fakeSWHID("cnt", 4)).setDirection(GraphDirection.BACKWARD).build());
+ assertEquals(11, actual.getCount());
+ }
+
+ @Test
+ public void backwardRevToRevRevToRel() {
+ CountResponse actual = client.countNodes(getTraversalRequestBuilder(fakeSWHID("rev", 3))
+ .setEdges("rev:rev,rev:rel").setDirection(GraphDirection.BACKWARD).build());
+ assertEquals(6, actual.getCount());
+ }
+
+ @Test
+ public void testWithEmptyMask() {
+ CountResponse actual = client.countNodes(
+ getTraversalRequestBuilder(fakeSWHID("dir", 12)).setMask(FieldMask.getDefaultInstance()).build());
+ assertEquals(8, actual.getCount());
+ }
+}
diff --git a/java/src/test/java/org/softwareheritage/graph/rpc/FindPathBetweenTest.java b/java/src/test/java/org/softwareheritage/graph/rpc/FindPathBetweenTest.java
new file mode 100644
--- /dev/null
+++ b/java/src/test/java/org/softwareheritage/graph/rpc/FindPathBetweenTest.java
@@ -0,0 +1,203 @@
+package org.softwareheritage.graph.rpc;
+
+import io.grpc.Status;
+import io.grpc.StatusRuntimeException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.softwareheritage.graph.SWHID;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class FindPathBetweenTest extends TraversalServiceTest {
+ private FindPathBetweenRequest.Builder getRequestBuilder(SWHID src, SWHID dst) {
+ return FindPathBetweenRequest.newBuilder().addSrc(src.toString()).addDst(dst.toString());
+ }
+
+ @Test
+ public void testSwhidErrors() {
+ StatusRuntimeException thrown;
+ thrown = assertThrows(StatusRuntimeException.class, () -> client
+ .findPathBetween(FindPathBetweenRequest.newBuilder().addSrc(fakeSWHID("cnt", 404).toString()).build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.findPathBetween(FindPathBetweenRequest
+ .newBuilder().addSrc("swh:1:lol:0000000000000000000000000000000000000001").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.findPathBetween(FindPathBetweenRequest
+ .newBuilder().addSrc("swh:1:cnt:000000000000000000000000000000000000000z").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class,
+ () -> client.findPathBetween(FindPathBetweenRequest.newBuilder().addSrc(TEST_ORIGIN_ID)
+ .addDst("swh:1:cnt:000000000000000000000000000000000000000z").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ }
+
+ @Test
+ public void testEdgeErrors() {
+ StatusRuntimeException thrown;
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.findPathBetween(FindPathBetweenRequest
+ .newBuilder().addSrc(TEST_ORIGIN_ID).addDst(TEST_ORIGIN_ID).setEdges("batracien:reptile").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ }
+
+ // Test path between ori 1 and cnt 4 (forward graph)
+ @Test
+ public void forwardRootToLeaf() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.findPathBetween(getRequestBuilder(new SWHID(TEST_ORIGIN_ID), fakeSWHID("cnt", 4)).build()));
+ List<SWHID> expected = List.of(new SWHID(TEST_ORIGIN_ID), fakeSWHID("snp", 20), fakeSWHID("rev", 9),
+ fakeSWHID("dir", 8), fakeSWHID("dir", 6), fakeSWHID("cnt", 4));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Test path between rev 18 and rev 3 (forward graph)
+ @Test
+ public void forwardRevToRev() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.findPathBetween(getRequestBuilder(fakeSWHID("rev", 18), fakeSWHID("rev", 3)).build()));
+ List<SWHID> expected = List.of(fakeSWHID("rev", 18), fakeSWHID("rev", 13), fakeSWHID("rev", 9),
+ fakeSWHID("rev", 3));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Test path between rev 3 and rev 18 (backward graph)
+ @Test
+ public void backwardRevToRev() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.findPathBetween(getRequestBuilder(fakeSWHID("rev", 3), fakeSWHID("rev", 18))
+ .setDirection(GraphDirection.BACKWARD).build()));
+ List<SWHID> expected = List.of(fakeSWHID("rev", 3), fakeSWHID("rev", 9), fakeSWHID("rev", 13),
+ fakeSWHID("rev", 18));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Test path between cnt 4 and itself (forward graph)
+ @Test
+ public void forwardCntToItself() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.findPathBetween(getRequestBuilder(fakeSWHID("cnt", 4), fakeSWHID("cnt", 4)).build()));
+ List<SWHID> expected = List.of(fakeSWHID("cnt", 4));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Start from ori and rel 19 and find cnt 14 or cnt 7 (forward graph)
+ @Test
+ public void forwardMultipleSourcesDest() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.findPathBetween(getRequestBuilder(fakeSWHID("rel", 19), fakeSWHID("cnt", 14))
+ .addSrc(TEST_ORIGIN_ID).addDst(fakeSWHID("cnt", 7).toString()).build()));
+ List<SWHID> expected = List.of(fakeSWHID("rel", 19), fakeSWHID("rev", 18), fakeSWHID("dir", 17),
+ fakeSWHID("cnt", 14));
+ }
+
+ // Start from cnt 4 and cnt 11 and find rev 13 or rev 9 (backward graph)
+ @Test
+ public void backwardMultipleSourcesDest() {
+ ArrayList<SWHID> actual = getSWHIDs(client.findPathBetween(
+ getRequestBuilder(fakeSWHID("cnt", 4), fakeSWHID("rev", 13)).setDirection(GraphDirection.BACKWARD)
+ .addSrc(fakeSWHID("cnt", 11).toString()).addDst(fakeSWHID("rev", 9).toString()).build()));
+ List<SWHID> expected = List.of(fakeSWHID("cnt", 11), fakeSWHID("dir", 12), fakeSWHID("rev", 13));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Start from all directories and find the origin (backward graph)
+ @Test
+ public void backwardMultipleSourcesAllDirToOri() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.findPathBetween(getRequestBuilder(fakeSWHID("dir", 2), new SWHID(TEST_ORIGIN_ID))
+ .addSrc(fakeSWHID("dir", 6).toString()).addSrc(fakeSWHID("dir", 8).toString())
+ .addSrc(fakeSWHID("dir", 12).toString()).addSrc(fakeSWHID("dir", 16).toString())
+ .addSrc(fakeSWHID("dir", 17).toString()).setDirection(GraphDirection.BACKWARD).build()));
+ List<SWHID> expected = List.of(fakeSWHID("dir", 8), fakeSWHID("rev", 9), fakeSWHID("snp", 20),
+ new SWHID(TEST_ORIGIN_ID));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Start from cnt 4 and find any rev (backward graph)
+ @Test
+ public void backwardCntToAnyRev() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.findPathBetween(getRequestBuilder(fakeSWHID("cnt", 4), fakeSWHID("rev", 3))
+ .addDst(fakeSWHID("rev", 9).toString()).addDst(fakeSWHID("rev", 13).toString())
+ .addDst(fakeSWHID("rev", 18).toString()).setDirection(GraphDirection.BACKWARD).build()));
+ List<SWHID> expected = List.of(fakeSWHID("cnt", 4), fakeSWHID("dir", 6), fakeSWHID("dir", 8),
+ fakeSWHID("rev", 9));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Impossible path between rev 9 and cnt 14
+ @Test
+ public void forwardImpossiblePath() {
+ StatusRuntimeException thrown = Assertions.assertThrows(StatusRuntimeException.class, () -> {
+ client.findPathBetween(getRequestBuilder(fakeSWHID("rev", 9), fakeSWHID("cnt", 14)).build());
+ });
+ Assertions.assertEquals(thrown.getStatus().getCode(), Status.NOT_FOUND.getCode());
+
+ // Reverse direction
+ thrown = Assertions.assertThrows(StatusRuntimeException.class, () -> {
+ client.findPathBetween(getRequestBuilder(fakeSWHID("cnt", 14), fakeSWHID("rev", 9))
+ .setDirection(GraphDirection.BACKWARD).build());
+ });
+ Assertions.assertEquals(thrown.getStatus().getCode(), Status.NOT_FOUND.getCode());
+ }
+
+ // Common ancestor between cnt 4 and cnt 15 : rev 18
+ @Test
+ public void commonAncestorBackwardBackward() {
+ Path p = client.findPathBetween(getRequestBuilder(fakeSWHID("cnt", 4), fakeSWHID("cnt", 15))
+ .setDirection(GraphDirection.BACKWARD).setDirectionReverse(GraphDirection.BACKWARD).build());
+ ArrayList<SWHID> actual = getSWHIDs(p);
+ SWHID expected = fakeSWHID("rev", 18);
+ Assertions.assertEquals(expected, actual.get(p.getMidpointIndex()));
+ }
+
+ // Common descendant between rev 13 and rev 3 : cnt 1 (with rev:dir,dir:dir,dir:cnt)
+ @Test
+ public void commonDescendantForwardForward() {
+ Path p = client.findPathBetween(
+ getRequestBuilder(fakeSWHID("rev", 13), fakeSWHID("rev", 3)).setDirection(GraphDirection.FORWARD)
+ .setDirectionReverse(GraphDirection.FORWARD).setEdges("rev:dir,dir:dir,dir:cnt").build());
+ ArrayList<SWHID> actual = getSWHIDs(p);
+ SWHID expected = fakeSWHID("cnt", 1);
+ Assertions.assertEquals(expected, actual.get(p.getMidpointIndex()));
+ }
+
+ // Path between rel 19 and cnt 15 with various max depths
+ @Test
+ public void maxDepth() {
+ // Works with max_depth = 2
+ ArrayList<SWHID> actual = getSWHIDs(client
+ .findPathBetween(getRequestBuilder(fakeSWHID("rel", 19), fakeSWHID("cnt", 15)).setMaxDepth(2).build()));
+ List<SWHID> expected = List.of(fakeSWHID("rel", 19), fakeSWHID("rev", 18), fakeSWHID("dir", 17),
+ fakeSWHID("dir", 16), fakeSWHID("cnt", 15));
+ Assertions.assertEquals(expected, actual);
+
+ // Check that it throws NOT_FOUND with max depth = 1
+ StatusRuntimeException thrown = Assertions.assertThrows(StatusRuntimeException.class, () -> {
+ client.findPathBetween(
+ getRequestBuilder(fakeSWHID("rel", 19), fakeSWHID("cnt", 15)).setMaxDepth(1).build());
+ });
+ Assertions.assertEquals(thrown.getStatus().getCode(), Status.NOT_FOUND.getCode());
+ }
+
+ // Path between rel 19 and cnt 15 with various max edges
+ @Test
+ public void maxEdges() {
+ // Works with max_edges = 3
+ ArrayList<SWHID> actual = getSWHIDs(client
+ .findPathBetween(getRequestBuilder(fakeSWHID("rel", 19), fakeSWHID("cnt", 15)).setMaxEdges(3).build()));
+ List<SWHID> expected = List.of(fakeSWHID("rel", 19), fakeSWHID("rev", 18), fakeSWHID("dir", 17),
+ fakeSWHID("dir", 16), fakeSWHID("cnt", 15));
+ Assertions.assertEquals(expected, actual);
+
+ // Check that it throws NOT_FOUND with max_edges = 2
+ StatusRuntimeException thrown = Assertions.assertThrows(StatusRuntimeException.class, () -> {
+ client.findPathBetween(
+ getRequestBuilder(fakeSWHID("rel", 19), fakeSWHID("cnt", 15)).setMaxEdges(2).build());
+ });
+ Assertions.assertEquals(thrown.getStatus().getCode(), Status.NOT_FOUND.getCode());
+ }
+}
diff --git a/java/src/test/java/org/softwareheritage/graph/rpc/FindPathToTest.java b/java/src/test/java/org/softwareheritage/graph/rpc/FindPathToTest.java
new file mode 100644
--- /dev/null
+++ b/java/src/test/java/org/softwareheritage/graph/rpc/FindPathToTest.java
@@ -0,0 +1,162 @@
+package org.softwareheritage.graph.rpc;
+
+import io.grpc.Status;
+import io.grpc.StatusRuntimeException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.softwareheritage.graph.SWHID;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class FindPathToTest extends TraversalServiceTest {
+ private FindPathToRequest.Builder getRequestBuilder(SWHID src, String allowedNodes) {
+ return FindPathToRequest.newBuilder().addSrc(src.toString())
+ .setTarget(NodeFilter.newBuilder().setTypes(allowedNodes).build());
+ }
+
+ @Test
+ public void testSrcErrors() {
+ StatusRuntimeException thrown;
+ thrown = assertThrows(StatusRuntimeException.class, () -> client
+ .findPathTo(FindPathToRequest.newBuilder().addSrc(fakeSWHID("cnt", 404).toString()).build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.findPathTo(
+ FindPathToRequest.newBuilder().addSrc("swh:1:lol:0000000000000000000000000000000000000001").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.findPathTo(
+ FindPathToRequest.newBuilder().addSrc("swh:1:cnt:000000000000000000000000000000000000000z").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ }
+
+ @Test
+ public void testEdgeErrors() {
+ StatusRuntimeException thrown;
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.findPathTo(
+ FindPathToRequest.newBuilder().addSrc(TEST_ORIGIN_ID).setEdges("batracien:reptile").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ }
+
+ @Test
+ public void testTargetErrors() {
+ StatusRuntimeException thrown;
+ thrown = assertThrows(StatusRuntimeException.class,
+ () -> client.findPathTo(FindPathToRequest.newBuilder().addSrc(TEST_ORIGIN_ID)
+ .setTarget(NodeFilter.newBuilder().setTypes("argoumante,eglomatique").build()).build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ }
+
+ // Test path between ori 1 and any dir (forward graph)
+ @Test
+ public void forwardOriToFirstDir() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.findPathTo(getRequestBuilder(new SWHID(TEST_ORIGIN_ID), "dir").build()));
+ List<SWHID> expected = List.of(new SWHID(TEST_ORIGIN_ID), fakeSWHID("snp", 20), fakeSWHID("rev", 9),
+ fakeSWHID("dir", 8));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Test path between rel 19 and any cnt (forward graph)
+ @Test
+ public void forwardRelToFirstCnt() {
+ ArrayList<SWHID> actual = getSWHIDs(client.findPathTo(getRequestBuilder(fakeSWHID("rel", 19), "cnt").build()));
+ List<SWHID> expected = List.of(fakeSWHID("rel", 19), fakeSWHID("rev", 18), fakeSWHID("dir", 17),
+ fakeSWHID("cnt", 14));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Test path between dir 16 and any rel (backward graph)
+ @Test
+ public void backwardDirToFirstRel() {
+ ArrayList<SWHID> actual = getSWHIDs(client.findPathTo(
+ getRequestBuilder(fakeSWHID("dir", 16), "rel").setDirection(GraphDirection.BACKWARD).build()));
+ List<SWHID> expected = List.of(fakeSWHID("dir", 16), fakeSWHID("dir", 17), fakeSWHID("rev", 18),
+ fakeSWHID("rel", 19));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Test path between cnt 4 and itself (forward graph)
+ @Test
+ public void forwardCntToItself() {
+ ArrayList<SWHID> actual = getSWHIDs(client.findPathTo(getRequestBuilder(fakeSWHID("cnt", 4), "cnt").build()));
+ List<SWHID> expected = List.of(fakeSWHID("cnt", 4));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Start from ori and rel 19 and find any cnt (forward graph)
+ @Test
+ public void forwardMultipleSources() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.findPathTo(getRequestBuilder(fakeSWHID("rel", 19), "cnt").addSrc(TEST_ORIGIN_ID).build()));
+ List<SWHID> expected = List.of(fakeSWHID("rel", 19), fakeSWHID("rev", 18), fakeSWHID("dir", 17),
+ fakeSWHID("cnt", 14));
+ }
+
+ // Start from cnt 4 and cnt 11 and find any rev (backward graph)
+ @Test
+ public void backwardMultipleSources() {
+ ArrayList<SWHID> actual = getSWHIDs(client.findPathTo(getRequestBuilder(fakeSWHID("cnt", 4), "rev")
+ .addSrc(fakeSWHID("cnt", 11).toString()).setDirection(GraphDirection.BACKWARD).build()));
+ List<SWHID> expected = List.of(fakeSWHID("cnt", 11), fakeSWHID("dir", 12), fakeSWHID("rev", 13));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Start from all directories and find any origin (backward graph)
+ @Test
+ public void backwardMultipleSourcesAllDirToOri() {
+ ArrayList<SWHID> actual = getSWHIDs(client.findPathTo(getRequestBuilder(fakeSWHID("dir", 2), "ori")
+ .addSrc(fakeSWHID("dir", 6).toString()).addSrc(fakeSWHID("dir", 8).toString())
+ .addSrc(fakeSWHID("dir", 12).toString()).addSrc(fakeSWHID("dir", 16).toString())
+ .addSrc(fakeSWHID("dir", 17).toString()).setDirection(GraphDirection.BACKWARD).build()));
+ List<SWHID> expected = List.of(fakeSWHID("dir", 8), fakeSWHID("rev", 9), fakeSWHID("snp", 20),
+ new SWHID(TEST_ORIGIN_ID));
+ Assertions.assertEquals(expected, actual);
+ }
+
+ // Impossible path between rev 9 and any release (forward graph)
+ @Test
+ public void forwardImpossiblePath() {
+ // Check that the return is STATUS.NOT_FOUND
+ StatusRuntimeException thrown = Assertions.assertThrows(StatusRuntimeException.class, () -> {
+ client.findPathTo(getRequestBuilder(fakeSWHID("rev", 9), "rel").build());
+ });
+ Assertions.assertEquals(thrown.getStatus(), Status.NOT_FOUND);
+ }
+
+ // Path from cnt 15 to any rel with various max depths
+ @Test
+ public void maxDepth() {
+ // Works with max_depth = 2
+ ArrayList<SWHID> actual = getSWHIDs(client.findPathTo(getRequestBuilder(fakeSWHID("cnt", 15), "rel")
+ .setDirection(GraphDirection.BACKWARD).setMaxDepth(4).build()));
+ List<SWHID> expected = List.of(fakeSWHID("cnt", 15), fakeSWHID("dir", 16), fakeSWHID("dir", 17),
+ fakeSWHID("rev", 18), fakeSWHID("rel", 19));
+ Assertions.assertEquals(expected, actual);
+
+ // Check that it throws NOT_FOUND with max depth = 1
+ StatusRuntimeException thrown = Assertions.assertThrows(StatusRuntimeException.class, () -> {
+ client.findPathTo(getRequestBuilder(fakeSWHID("cnt", 15), "rel").setDirection(GraphDirection.BACKWARD)
+ .setMaxDepth(3).build());
+ });
+ Assertions.assertEquals(thrown.getStatus().getCode(), Status.NOT_FOUND.getCode());
+ }
+
+ // Path from cnt 15 to any rel with various max edges
+ @Test
+ public void maxEdges() {
+ ArrayList<SWHID> actual = getSWHIDs(client.findPathTo(getRequestBuilder(fakeSWHID("cnt", 15), "rel")
+ .setDirection(GraphDirection.BACKWARD).setMaxEdges(4).build()));
+ List<SWHID> expected = List.of(fakeSWHID("cnt", 15), fakeSWHID("dir", 16), fakeSWHID("dir", 17),
+ fakeSWHID("rev", 18), fakeSWHID("rel", 19));
+ Assertions.assertEquals(expected, actual);
+
+ StatusRuntimeException thrown = Assertions.assertThrows(StatusRuntimeException.class, () -> {
+ client.findPathTo(getRequestBuilder(fakeSWHID("cnt", 15), "rel").setDirection(GraphDirection.BACKWARD)
+ .setMaxEdges(3).build());
+ });
+ Assertions.assertEquals(thrown.getStatus().getCode(), Status.NOT_FOUND.getCode());
+ }
+}
diff --git a/java/src/test/java/org/softwareheritage/graph/rpc/GetNodeTest.java b/java/src/test/java/org/softwareheritage/graph/rpc/GetNodeTest.java
new file mode 100644
--- /dev/null
+++ b/java/src/test/java/org/softwareheritage/graph/rpc/GetNodeTest.java
@@ -0,0 +1,284 @@
+package org.softwareheritage.graph.rpc;
+
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.FieldMask;
+import io.grpc.Status;
+import io.grpc.StatusRuntimeException;
+import org.junit.jupiter.api.Test;
+import org.softwareheritage.graph.SWHID;
+
+import java.util.*;
+
+import static org.junit.jupiter.api.Assertions.*;
+
+public class GetNodeTest extends TraversalServiceTest {
+ @Test
+ public void testNotFound() {
+ StatusRuntimeException thrown = assertThrows(StatusRuntimeException.class,
+ () -> client.getNode(GetNodeRequest.newBuilder().setSwhid(fakeSWHID("cnt", 404).toString()).build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ }
+
+ @Test
+ public void testInvalidSwhid() {
+ StatusRuntimeException thrown;
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.getNode(
+ GetNodeRequest.newBuilder().setSwhid("swh:1:lol:0000000000000000000000000000000000000001").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class, () -> client.getNode(
+ GetNodeRequest.newBuilder().setSwhid("swh:1:cnt:000000000000000000000000000000000000000z").build()));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ }
+
+ @Test
+ public void testContents() {
+ List<Integer> expectedCnts = List.of(1, 4, 5, 7, 11, 14, 15);
+ Map<Integer, Integer> expectedLengths = Map.of(1, 42, 4, 404, 5, 1337, 7, 666, 11, 313, 14, 14, 15, 404);
+ Set<Integer> expectedSkipped = Set.of(15);
+
+ for (Integer cntId : expectedCnts) {
+ Node n = client.getNode(GetNodeRequest.newBuilder().setSwhid(fakeSWHID("cnt", cntId).toString()).build());
+ assertTrue(n.hasCnt());
+ assertTrue(n.getCnt().hasLength());
+ assertEquals((long) expectedLengths.get(cntId), n.getCnt().getLength());
+ assertTrue(n.getCnt().hasIsSkipped());
+ assertEquals(expectedSkipped.contains(cntId), n.getCnt().getIsSkipped());
+ }
+ }
+
+ @Test
+ public void testRevisions() {
+ List<Integer> expectedRevs = List.of(3, 9, 13, 18);
+ Map<Integer, String> expectedMessages = Map.of(3, "Initial commit", 9, "Add parser", 13, "Add tests", 18,
+ "Refactor codebase");
+
+ Map<Integer, String> expectedAuthors = Map.of(3, "foo", 9, "bar", 13, "foo", 18, "baz");
+ Map<Integer, String> expectedCommitters = Map.of(3, "foo", 9, "bar", 13, "bar", 18, "foo");
+
+ Map<Integer, Long> expectedAuthorTimestamps = Map.of(3, 1111122220L, 9, 1111144440L, 13, 1111166660L, 18,
+ 1111177770L);
+ Map<Integer, Long> expectedCommitterTimestamps = Map.of(3, 1111122220L, 9, 1111155550L, 13, 1111166660L, 18,
+ 1111177770L);
+ Map<Integer, Integer> expectedAuthorTimestampOffsets = Map.of(3, 120, 9, 120, 13, 120, 18, 0);
+ Map<Integer, Integer> expectedCommitterTimestampOffsets = Map.of(3, 120, 9, 120, 13, 120, 18, 0);
+
+ HashMap<Integer, String> personMapping = new HashMap<>();
+ for (Integer revId : expectedRevs) {
+ Node n = client.getNode(GetNodeRequest.newBuilder().setSwhid(fakeSWHID("rev", revId).toString()).build());
+ assertTrue(n.hasRev());
+ assertTrue(n.getRev().hasMessage());
+ assertEquals(expectedMessages.get(revId), n.getRev().getMessage().toStringUtf8());
+
+ // Persons are anonymized, we just need to check that the mapping is self-consistent
+ assertTrue(n.getRev().hasAuthor());
+ assertTrue(n.getRev().hasCommitter());
+ int[] actualPersons = new int[]{(int) n.getRev().getAuthor(), (int) n.getRev().getCommitter()};
+ String[] expectedPersons = new String[]{expectedAuthors.get(revId), expectedCommitters.get(revId)};
+ for (int i = 0; i < actualPersons.length; i++) {
+ int actualPerson = actualPersons[i];
+ String expectedPerson = expectedPersons[i];
+ assertTrue(actualPerson >= 0);
+ if (personMapping.containsKey(actualPerson)) {
+ assertEquals(personMapping.get(actualPerson), expectedPerson);
+ } else {
+ personMapping.put(actualPerson, expectedPerson);
+ }
+ }
+
+ assertTrue(n.getRev().hasAuthorDate());
+ assertTrue(n.getRev().hasAuthorDateOffset());
+ assertTrue(n.getRev().hasCommitterDate());
+ assertTrue(n.getRev().hasCommitterDateOffset());
+
+ // FIXME: all the timestamps are one hour off?!
+ // System.err.println(revId + " " + n.getRev().getAuthorDate() + " " +
+ // n.getRev().getAuthorDateOffset());
+ // System.err.println(revId + " " + n.getRev().getCommitterDate() + " " +
+ // n.getRev().getCommitterDateOffset());
+
+ // assertEquals(expectedAuthorTimestamps.get(revId), n.getRev().getAuthorDate());
+ assertEquals(expectedAuthorTimestampOffsets.get(revId), n.getRev().getAuthorDateOffset());
+ // assertEquals(expectedCommitterTimestamps.get(revId), n.getRev().getAuthorDate());
+ assertEquals(expectedCommitterTimestampOffsets.get(revId), n.getRev().getAuthorDateOffset());
+ }
+ }
+
+ @Test
+ public void testReleases() {
+ List<Integer> expectedRels = List.of(10, 19);
+ Map<Integer, String> expectedMessages = Map.of(10, "Version 1.0", 19, "Version 2.0");
+ Map<Integer, String> expectedNames = Map.of(10, "v1.0", 19, "v2.0");
+
+ Map<Integer, String> expectedAuthors = Map.of(10, "foo", 19, "bar");
+
+ Map<Integer, Long> expectedAuthorTimestamps = Map.of(10, 1234567890L);
+ Map<Integer, Integer> expectedAuthorTimestampOffsets = Map.of(3, 120);
+
+ HashMap<Integer, String> personMapping = new HashMap<>();
+ for (Integer relId : expectedRels) {
+ Node n = client.getNode(GetNodeRequest.newBuilder().setSwhid(fakeSWHID("rel", relId).toString()).build());
+ assertTrue(n.hasRel());
+ assertTrue(n.getRel().hasMessage());
+ assertEquals(expectedMessages.get(relId), n.getRel().getMessage().toStringUtf8());
+ // FIXME: names are always empty?!
+ // System.err.println(relId + " " + n.getRel().getName());
+ // assertEquals(expectedNames.get(relId), n.getRel().getName().toStringUtf8());
+
+ // Persons are anonymized, we just need to check that the mapping is self-consistent
+ assertTrue(n.getRel().hasAuthor());
+ int actualPerson = (int) n.getRel().getAuthor();
+ String expectedPerson = expectedAuthors.get(relId);
+ assertTrue(actualPerson >= 0);
+ if (personMapping.containsKey(actualPerson)) {
+ assertEquals(personMapping.get(actualPerson), expectedPerson);
+ } else {
+ personMapping.put(actualPerson, expectedPerson);
+ }
+
+ assertTrue(n.getRel().hasAuthorDate());
+ assertTrue(n.getRel().hasAuthorDateOffset());
+
+ // FIXME: all the timestamps are one hour off?!
+ // if (expectedAuthorTimestamps.containsKey(relId)) {
+ // assertEquals(expectedAuthorTimestamps.get(revId), n.getRev().getAuthorDate());
+ // }
+ if (expectedAuthorTimestampOffsets.containsKey(relId)) {
+ assertEquals(expectedAuthorTimestampOffsets.get(relId), n.getRev().getAuthorDateOffset());
+ }
+ }
+ }
+
+ @Test
+ public void testOrigins() {
+ List<SWHID> expectedOris = List.of(new SWHID(TEST_ORIGIN_ID));
+ Map<SWHID, String> expectedUrls = Map.of(new SWHID(TEST_ORIGIN_ID), "https://example.com/swh/graph");
+
+ for (SWHID oriSwhid : expectedOris) {
+ Node n = client.getNode(GetNodeRequest.newBuilder().setSwhid(oriSwhid.toString()).build());
+ assertTrue(n.hasOri());
+ assertTrue(n.getOri().hasUrl());
+ assertEquals(expectedUrls.get(oriSwhid), n.getOri().getUrl());
+ }
+ }
+
+ @Test
+ public void testCntMask() {
+ Node n;
+ String swhid = fakeSWHID("cnt", 1).toString();
+
+ // No mask, all fields present
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid).build());
+ assertTrue(n.hasCnt());
+ assertTrue(n.getCnt().hasLength());
+ assertEquals(42, n.getCnt().getLength());
+ assertTrue(n.getCnt().hasIsSkipped());
+ assertFalse(n.getCnt().getIsSkipped());
+
+ // Empty mask, no fields present
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid).setMask(FieldMask.getDefaultInstance()).build());
+ assertFalse(n.getCnt().hasLength());
+ assertFalse(n.getCnt().hasIsSkipped());
+
+ // Mask with length, no isSkipped
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid)
+ .setMask(FieldMask.newBuilder().addPaths("cnt.length").build()).build());
+ assertTrue(n.getCnt().hasLength());
+ assertFalse(n.getCnt().hasIsSkipped());
+
+ // Mask with isSkipped, no length
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid)
+ .setMask(FieldMask.newBuilder().addPaths("cnt.is_skipped").build()).build());
+ assertFalse(n.getCnt().hasLength());
+ assertTrue(n.getCnt().hasIsSkipped());
+ }
+
+ @Test
+ public void testRevMask() {
+ Node n;
+ String swhid = fakeSWHID("rev", 3).toString();
+
+ // No mask, all fields present
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid).build());
+ assertTrue(n.hasRev());
+ assertTrue(n.getRev().hasMessage());
+ assertTrue(n.getRev().hasAuthor());
+ assertTrue(n.getRev().hasAuthorDate());
+ assertTrue(n.getRev().hasAuthorDateOffset());
+ assertTrue(n.getRev().hasCommitter());
+ assertTrue(n.getRev().hasCommitterDate());
+ assertTrue(n.getRev().hasCommitterDateOffset());
+
+ // Empty mask, no fields present
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid).setMask(FieldMask.getDefaultInstance()).build());
+ assertFalse(n.getRev().hasMessage());
+ assertFalse(n.getRev().hasAuthor());
+ assertFalse(n.getRev().hasAuthorDate());
+ assertFalse(n.getRev().hasAuthorDateOffset());
+ assertFalse(n.getRev().hasCommitter());
+ assertFalse(n.getRev().hasCommitterDate());
+ assertFalse(n.getRev().hasCommitterDateOffset());
+
+ // Test all masks with single fields
+ for (Descriptors.FieldDescriptor includedField : RevisionData.getDefaultInstance().getAllFields().keySet()) {
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid)
+ .setMask(FieldMask.newBuilder().addPaths("rev." + includedField.getName()).build()).build());
+ for (Descriptors.FieldDescriptor f : n.getRev().getDescriptorForType().getFields()) {
+ assertEquals(n.getRev().hasField(f), f.getName().equals(includedField.getName()));
+ }
+ }
+ }
+
+ @Test
+ public void testRelMask() {
+ Node n;
+ String swhid = fakeSWHID("rel", 19).toString();
+
+ // No mask, all fields present
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid).build());
+ assertTrue(n.hasRel());
+ assertTrue(n.getRel().hasMessage());
+ assertTrue(n.getRel().hasAuthor());
+ assertTrue(n.getRel().hasAuthorDate());
+ assertTrue(n.getRel().hasAuthorDateOffset());
+
+ // Empty mask, no fields present
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid).setMask(FieldMask.getDefaultInstance()).build());
+ assertFalse(n.getRel().hasMessage());
+ assertFalse(n.getRel().hasAuthor());
+ assertFalse(n.getRel().hasAuthorDate());
+ assertFalse(n.getRel().hasAuthorDateOffset());
+
+ // Test all masks with single fields
+ for (Descriptors.FieldDescriptor includedField : ReleaseData.getDefaultInstance().getAllFields().keySet()) {
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid)
+ .setMask(FieldMask.newBuilder().addPaths("rel." + includedField.getName()).build()).build());
+ for (Descriptors.FieldDescriptor f : n.getRel().getDescriptorForType().getFields()) {
+ assertEquals(n.getRel().hasField(f), f.getName().equals(includedField.getName()));
+ }
+ }
+ }
+
+ @Test
+ public void testOriMask() {
+ Node n;
+ String swhid = TEST_ORIGIN_ID;
+
+ // No mask, all fields present
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid).build());
+ assertTrue(n.hasOri());
+ assertTrue(n.getOri().hasUrl());
+
+ // Empty mask, no fields present
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid).setMask(FieldMask.getDefaultInstance()).build());
+ assertFalse(n.getOri().hasUrl());
+
+ // Test all masks with single fields
+ for (Descriptors.FieldDescriptor includedField : OriginData.getDefaultInstance().getAllFields().keySet()) {
+ n = client.getNode(GetNodeRequest.newBuilder().setSwhid(swhid)
+ .setMask(FieldMask.newBuilder().addPaths("ori." + includedField.getName()).build()).build());
+ for (Descriptors.FieldDescriptor f : n.getOri().getDescriptorForType().getFields()) {
+ assertEquals(n.getOri().hasField(f), f.getName().equals(includedField.getName()));
+ }
+ }
+ }
+}
diff --git a/java/src/test/java/org/softwareheritage/graph/rpc/StatsTest.java b/java/src/test/java/org/softwareheritage/graph/rpc/StatsTest.java
new file mode 100644
--- /dev/null
+++ b/java/src/test/java/org/softwareheritage/graph/rpc/StatsTest.java
@@ -0,0 +1,18 @@
+package org.softwareheritage.graph.rpc;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.*;
+
+public class StatsTest extends TraversalServiceTest {
+ @Test
+ public void testStats() {
+ StatsResponse stats = client.stats(StatsRequest.getDefaultInstance());
+ assertEquals(stats.getNumNodes(), 21);
+ assertEquals(stats.getNumEdges(), 23);
+ assertEquals(stats.getIndegreeMin(), 0);
+ assertEquals(stats.getIndegreeMax(), 3);
+ assertEquals(stats.getOutdegreeMin(), 0);
+ assertEquals(stats.getOutdegreeMax(), 3);
+ }
+}
diff --git a/java/src/test/java/org/softwareheritage/graph/rpc/TraversalServiceTest.java b/java/src/test/java/org/softwareheritage/graph/rpc/TraversalServiceTest.java
new file mode 100644
--- /dev/null
+++ b/java/src/test/java/org/softwareheritage/graph/rpc/TraversalServiceTest.java
@@ -0,0 +1,58 @@
+package org.softwareheritage.graph.rpc;
+
+import io.grpc.ManagedChannel;
+import io.grpc.Server;
+import io.grpc.inprocess.InProcessChannelBuilder;
+import io.grpc.inprocess.InProcessServerBuilder;
+import io.grpc.testing.GrpcCleanupRule;
+import org.junit.Rule;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.softwareheritage.graph.GraphTest;
+import org.softwareheritage.graph.SWHID;
+import org.softwareheritage.graph.SwhBidirectionalGraph;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+
+public class TraversalServiceTest extends GraphTest {
+ @Rule
+ public final GrpcCleanupRule grpcCleanup = new GrpcCleanupRule();
+
+ private static Server server;
+ private static ManagedChannel channel;
+ protected static SwhBidirectionalGraph g;
+ protected static TraversalServiceGrpc.TraversalServiceBlockingStub client;
+
+ @BeforeAll
+ static void setup() throws Exception {
+ String serverName = InProcessServerBuilder.generateName();
+ g = GraphServer.loadGraph(getGraphPath().toString());
+ server = InProcessServerBuilder.forName(serverName).directExecutor()
+ .addService(new GraphServer.TraversalService(g.copy())).build().start();
+ channel = InProcessChannelBuilder.forName(serverName).directExecutor().build();
+ client = TraversalServiceGrpc.newBlockingStub(channel);
+ }
+
+ @AfterAll
+ static void teardown() {
+ channel.shutdownNow();
+ server.shutdownNow();
+ }
+
+ public ArrayList<SWHID> getSWHIDs(Iterator<Node> it) {
+ ArrayList<SWHID> res = new ArrayList<>();
+ it.forEachRemaining((Node n) -> {
+ res.add(new SWHID(n.getSwhid()));
+ });
+ return res;
+ }
+
+ public ArrayList<SWHID> getSWHIDs(Path p) {
+ ArrayList<SWHID> res = new ArrayList<>();
+ p.getNodeList().forEach((Node n) -> {
+ res.add(new SWHID(n.getSwhid()));
+ });
+ return res;
+ }
+}
diff --git a/java/src/test/java/org/softwareheritage/graph/LeavesTest.java b/java/src/test/java/org/softwareheritage/graph/rpc/TraverseLeavesTest.java
rename from java/src/test/java/org/softwareheritage/graph/LeavesTest.java
rename to java/src/test/java/org/softwareheritage/graph/rpc/TraverseLeavesTest.java
--- a/java/src/test/java/org/softwareheritage/graph/LeavesTest.java
+++ b/java/src/test/java/org/softwareheritage/graph/rpc/TraverseLeavesTest.java
@@ -1,18 +1,20 @@
-package org.softwareheritage.graph;
+package org.softwareheritage.graph.rpc;
+
+import org.junit.jupiter.api.Test;
+import org.softwareheritage.graph.GraphTest;
+import org.softwareheritage.graph.SWHID;
import java.util.ArrayList;
-import org.junit.jupiter.api.Test;
-import org.softwareheritage.graph.server.Endpoint;
+public class TraverseLeavesTest extends TraversalServiceTest {
+ private TraversalRequest.Builder getLeavesRequestBuilder(SWHID src) {
+ return TraversalRequest.newBuilder().addSrc(src.toString())
+ .setReturnNodes(NodeFilter.newBuilder().setMaxTraversalSuccessors(0).build());
+ }
-// Avoid warnings concerning Endpoint.Output.result manual cast
-@SuppressWarnings("unchecked")
-public class LeavesTest extends GraphTest {
@Test
public void forwardFromSnp() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:snp:0000000000000000000000000000000000000020");
- Endpoint endpoint = new Endpoint(graph, "forward", "*");
+ TraversalRequest request = getLeavesRequestBuilder(fakeSWHID("snp", 20)).build();
ArrayList<SWHID> expectedLeaves = new ArrayList<>();
expectedLeaves.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000001"));
@@ -20,16 +22,14 @@
expectedLeaves.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000005"));
expectedLeaves.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000007"));
- ArrayList<SWHID> actualLeaves = (ArrayList) endpoint.leaves(new Endpoint.Input(src)).result;
+ ArrayList<SWHID> actualLeaves = getSWHIDs(client.traverse(request));
GraphTest.assertEqualsAnyOrder(expectedLeaves, actualLeaves);
}
@Test
public void forwardFromRel() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:rel:0000000000000000000000000000000000000019");
- Endpoint endpoint = new Endpoint(graph, "forward", "*");
-
+ TraversalRequest request = getLeavesRequestBuilder(fakeSWHID("rel", 19)).build();
+ ArrayList<SWHID> actualLeaves = getSWHIDs(client.traverse(request));
ArrayList<SWHID> expectedLeaves = new ArrayList<>();
expectedLeaves.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000015"));
expectedLeaves.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000014"));
@@ -39,69 +39,55 @@
expectedLeaves.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000007"));
expectedLeaves.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000011"));
- ArrayList<SWHID> actualLeaves = (ArrayList) endpoint.leaves(new Endpoint.Input(src)).result;
GraphTest.assertEqualsAnyOrder(expectedLeaves, actualLeaves);
}
@Test
public void backwardFromLeaf() {
- SwhBidirectionalGraph graph = getGraph();
-
- Endpoint endpoint1 = new Endpoint(graph, "backward", "*");
- SWHID src1 = new SWHID("swh:1:cnt:0000000000000000000000000000000000000015");
+ TraversalRequest request1 = getLeavesRequestBuilder(fakeSWHID("cnt", 15)).setDirection(GraphDirection.BACKWARD)
+ .build();
+ ArrayList<SWHID> actualLeaves1 = getSWHIDs(client.traverse(request1));
ArrayList<SWHID> expectedLeaves1 = new ArrayList<>();
expectedLeaves1.add(new SWHID("swh:1:rel:0000000000000000000000000000000000000019"));
- ArrayList<SWHID> actualLeaves1 = (ArrayList) endpoint1.leaves(new Endpoint.Input(src1)).result;
GraphTest.assertEqualsAnyOrder(expectedLeaves1, actualLeaves1);
- Endpoint endpoint2 = new Endpoint(graph, "backward", "*");
- SWHID src2 = new SWHID("swh:1:cnt:0000000000000000000000000000000000000004");
+ TraversalRequest request2 = getLeavesRequestBuilder(fakeSWHID("cnt", 4)).setDirection(GraphDirection.BACKWARD)
+ .build();
+ ArrayList<SWHID> actualLeaves2 = getSWHIDs(client.traverse(request2));
ArrayList<SWHID> expectedLeaves2 = new ArrayList<>();
expectedLeaves2.add(new SWHID(TEST_ORIGIN_ID));
expectedLeaves2.add(new SWHID("swh:1:rel:0000000000000000000000000000000000000019"));
- ArrayList<SWHID> actualLeaves2 = (ArrayList) endpoint2.leaves(new Endpoint.Input(src2)).result;
GraphTest.assertEqualsAnyOrder(expectedLeaves2, actualLeaves2);
}
@Test
public void forwardRevToRevOnly() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:rev:0000000000000000000000000000000000000018");
- Endpoint endpoint = new Endpoint(graph, "forward", "rev:rev");
-
+ TraversalRequest request = getLeavesRequestBuilder(fakeSWHID("rev", 18)).setEdges("rev:rev").build();
+ ArrayList<SWHID> actualLeaves = getSWHIDs(client.traverse(request));
ArrayList<SWHID> expectedLeaves = new ArrayList<>();
expectedLeaves.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000003"));
-
- ArrayList<SWHID> actualLeaves = (ArrayList) endpoint.leaves(new Endpoint.Input(src)).result;
GraphTest.assertEqualsAnyOrder(expectedLeaves, actualLeaves);
}
@Test
public void forwardDirToAll() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:dir:0000000000000000000000000000000000000008");
- Endpoint endpoint = new Endpoint(graph, "forward", "dir:*");
-
+ TraversalRequest request = getLeavesRequestBuilder(fakeSWHID("dir", 8)).setEdges("dir:*").build();
+ ArrayList<SWHID> actualLeaves = getSWHIDs(client.traverse(request));
ArrayList<SWHID> expectedLeaves = new ArrayList<>();
expectedLeaves.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000004"));
expectedLeaves.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000005"));
expectedLeaves.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000001"));
expectedLeaves.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000007"));
-
- ArrayList<SWHID> actualLeaves = (ArrayList) endpoint.leaves(new Endpoint.Input(src)).result;
GraphTest.assertEqualsAnyOrder(expectedLeaves, actualLeaves);
}
@Test
public void backwardCntToDirDirToDir() {
- SwhBidirectionalGraph graph = getGraph();
- SWHID src = new SWHID("swh:1:cnt:0000000000000000000000000000000000000005");
- Endpoint endpoint = new Endpoint(graph, "backward", "cnt:dir,dir:dir");
-
+ TraversalRequest request = getLeavesRequestBuilder(fakeSWHID("cnt", 5)).setEdges("cnt:dir,dir:dir")
+ .setDirection(GraphDirection.BACKWARD).build();
+ ArrayList<SWHID> actualLeaves = getSWHIDs(client.traverse(request));
ArrayList<SWHID> expectedLeaves = new ArrayList<>();
expectedLeaves.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000012"));
-
- ArrayList<SWHID> actualLeaves = (ArrayList) endpoint.leaves(new Endpoint.Input(src)).result;
GraphTest.assertEqualsAnyOrder(expectedLeaves, actualLeaves);
}
}
diff --git a/java/src/test/java/org/softwareheritage/graph/rpc/TraverseNeighborsTest.java b/java/src/test/java/org/softwareheritage/graph/rpc/TraverseNeighborsTest.java
new file mode 100644
--- /dev/null
+++ b/java/src/test/java/org/softwareheritage/graph/rpc/TraverseNeighborsTest.java
@@ -0,0 +1,130 @@
+package org.softwareheritage.graph.rpc;
+
+import org.junit.jupiter.api.Test;
+import org.softwareheritage.graph.GraphTest;
+import org.softwareheritage.graph.SWHID;
+
+import java.util.ArrayList;
+
+public class TraverseNeighborsTest extends TraversalServiceTest {
+ private TraversalRequest.Builder getNeighborsRequestBuilder(SWHID src) {
+ return TraversalRequest.newBuilder().addSrc(src.toString()).setMinDepth(1).setMaxDepth(1);
+ }
+
+ @Test
+ public void zeroNeighbor() {
+ ArrayList<SWHID> expectedNodes = new ArrayList<>();
+
+ TraversalRequest request1 = getNeighborsRequestBuilder(new SWHID(TEST_ORIGIN_ID))
+ .setDirection(GraphDirection.BACKWARD).build();
+ ArrayList<SWHID> actuals1 = getSWHIDs(client.traverse(request1));
+ GraphTest.assertEqualsAnyOrder(expectedNodes, actuals1);
+
+ TraversalRequest request2 = getNeighborsRequestBuilder(fakeSWHID("cnt", 4)).build();
+ ArrayList<SWHID> actuals2 = getSWHIDs(client.traverse(request2));
+ GraphTest.assertEqualsAnyOrder(expectedNodes, actuals2);
+
+ TraversalRequest request3 = getNeighborsRequestBuilder(fakeSWHID("cnt", 15)).build();
+ ArrayList<SWHID> actuals3 = getSWHIDs(client.traverse(request3));
+ GraphTest.assertEqualsAnyOrder(expectedNodes, actuals3);
+
+ TraversalRequest request4 = getNeighborsRequestBuilder(fakeSWHID("rel", 19))
+ .setDirection(GraphDirection.BACKWARD).build();
+ ArrayList<SWHID> actuals4 = getSWHIDs(client.traverse(request4));
+ GraphTest.assertEqualsAnyOrder(expectedNodes, actuals4);
+
+ TraversalRequest request5 = getNeighborsRequestBuilder(fakeSWHID("dir", 8)).setEdges("snp:*,rev:*,rel:*")
+ .build();
+ ArrayList<SWHID> actuals5 = getSWHIDs(client.traverse(request5));
+ GraphTest.assertEqualsAnyOrder(expectedNodes, actuals5);
+ }
+
+ @Test
+ public void oneNeighbor() {
+ TraversalRequest request1 = getNeighborsRequestBuilder(fakeSWHID("rev", 3)).build();
+ ArrayList<SWHID> actuals1 = getSWHIDs(client.traverse(request1));
+ ArrayList<SWHID> expectedNodes1 = new ArrayList<>();
+ expectedNodes1.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000002"));
+ GraphTest.assertEqualsAnyOrder(expectedNodes1, actuals1);
+
+ TraversalRequest request2 = getNeighborsRequestBuilder(fakeSWHID("dir", 17)).setEdges("dir:cnt").build();
+ ArrayList<SWHID> actuals2 = getSWHIDs(client.traverse(request2));
+ ArrayList<SWHID> expectedNodes2 = new ArrayList<>();
+ expectedNodes2.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000014"));
+ GraphTest.assertEqualsAnyOrder(expectedNodes2, actuals2);
+
+ TraversalRequest request3 = getNeighborsRequestBuilder(fakeSWHID("dir", 12))
+ .setDirection(GraphDirection.BACKWARD).build();
+ ArrayList<SWHID> actuals3 = getSWHIDs(client.traverse(request3));
+ ArrayList<SWHID> expectedNodes3 = new ArrayList<>();
+ expectedNodes3.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000013"));
+ GraphTest.assertEqualsAnyOrder(expectedNodes3, actuals3);
+
+ TraversalRequest request4 = getNeighborsRequestBuilder(fakeSWHID("rev", 9))
+ .setDirection(GraphDirection.BACKWARD).setEdges("rev:rev").build();
+ ArrayList<SWHID> actuals4 = getSWHIDs(client.traverse(request4));
+ ArrayList<SWHID> expectedNodes4 = new ArrayList<>();
+ expectedNodes4.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000013"));
+ GraphTest.assertEqualsAnyOrder(expectedNodes4, actuals4);
+
+ TraversalRequest request5 = getNeighborsRequestBuilder(fakeSWHID("snp", 20))
+ .setDirection(GraphDirection.BACKWARD).build();
+ ArrayList<SWHID> actuals5 = getSWHIDs(client.traverse(request5));
+ ArrayList<SWHID> expectedNodes5 = new ArrayList<>();
+ expectedNodes5.add(new SWHID(TEST_ORIGIN_ID));
+ GraphTest.assertEqualsAnyOrder(expectedNodes5, actuals5);
+ }
+
+ @Test
+ public void twoNeighbors() {
+ TraversalRequest request1 = getNeighborsRequestBuilder(fakeSWHID("snp", 20)).build();
+ ArrayList<SWHID> actuals1 = getSWHIDs(client.traverse(request1));
+ ArrayList<SWHID> expectedNodes1 = new ArrayList<>();
+ expectedNodes1.add(new SWHID("swh:1:rel:0000000000000000000000000000000000000010"));
+ expectedNodes1.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000009"));
+ GraphTest.assertEqualsAnyOrder(expectedNodes1, actuals1);
+
+ TraversalRequest request2 = getNeighborsRequestBuilder(fakeSWHID("dir", 8)).setEdges("dir:cnt").build();
+ ArrayList<SWHID> actuals2 = getSWHIDs(client.traverse(request2));
+ ArrayList<SWHID> expectedNodes2 = new ArrayList<>();
+ expectedNodes2.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000001"));
+ expectedNodes2.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000007"));
+ GraphTest.assertEqualsAnyOrder(expectedNodes2, actuals2);
+
+ TraversalRequest request3 = getNeighborsRequestBuilder(fakeSWHID("cnt", 1))
+ .setDirection(GraphDirection.BACKWARD).build();
+ ArrayList<SWHID> actuals3 = getSWHIDs(client.traverse(request3));
+ ArrayList<SWHID> expectedNodes3 = new ArrayList<>();
+ expectedNodes3.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000008"));
+ expectedNodes3.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000002"));
+ GraphTest.assertEqualsAnyOrder(expectedNodes3, actuals3);
+
+ TraversalRequest request4 = getNeighborsRequestBuilder(fakeSWHID("rev", 9))
+ .setDirection(GraphDirection.BACKWARD).setEdges("rev:snp,rev:rel").build();
+ ArrayList<SWHID> actuals4 = getSWHIDs(client.traverse(request4));
+ ArrayList<SWHID> expectedNodes4 = new ArrayList<>();
+ expectedNodes4.add(new SWHID("swh:1:snp:0000000000000000000000000000000000000020"));
+ expectedNodes4.add(new SWHID("swh:1:rel:0000000000000000000000000000000000000010"));
+ GraphTest.assertEqualsAnyOrder(expectedNodes4, actuals4);
+ }
+
+ @Test
+ public void threeNeighbors() {
+ TraversalRequest request1 = getNeighborsRequestBuilder(fakeSWHID("dir", 8)).build();
+ ArrayList<SWHID> actuals1 = getSWHIDs(client.traverse(request1));
+ ArrayList<SWHID> expectedNodes1 = new ArrayList<>();
+ expectedNodes1.add(new SWHID("swh:1:dir:0000000000000000000000000000000000000006"));
+ expectedNodes1.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000001"));
+ expectedNodes1.add(new SWHID("swh:1:cnt:0000000000000000000000000000000000000007"));
+ GraphTest.assertEqualsAnyOrder(expectedNodes1, actuals1);
+
+ TraversalRequest request2 = getNeighborsRequestBuilder(fakeSWHID("rev", 9))
+ .setDirection(GraphDirection.BACKWARD).build();
+ ArrayList<SWHID> actuals2 = getSWHIDs(client.traverse(request2));
+ ArrayList<SWHID> expectedNodes2 = new ArrayList<>();
+ expectedNodes2.add(new SWHID("swh:1:snp:0000000000000000000000000000000000000020"));
+ expectedNodes2.add(new SWHID("swh:1:rel:0000000000000000000000000000000000000010"));
+ expectedNodes2.add(new SWHID("swh:1:rev:0000000000000000000000000000000000000013"));
+ GraphTest.assertEqualsAnyOrder(expectedNodes2, actuals2);
+ }
+}
diff --git a/java/src/test/java/org/softwareheritage/graph/rpc/TraverseNodesPropertiesTest.java b/java/src/test/java/org/softwareheritage/graph/rpc/TraverseNodesPropertiesTest.java
new file mode 100644
--- /dev/null
+++ b/java/src/test/java/org/softwareheritage/graph/rpc/TraverseNodesPropertiesTest.java
@@ -0,0 +1,110 @@
+package org.softwareheritage.graph.rpc;
+
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.FieldMask;
+import com.google.protobuf.Message;
+import it.unimi.dsi.big.webgraph.labelling.ArcLabelledNodeIterator;
+import org.junit.jupiter.api.Test;
+import org.softwareheritage.graph.SWHID;
+import org.softwareheritage.graph.SwhUnidirectionalGraph;
+import org.softwareheritage.graph.labels.DirEntry;
+
+import java.util.*;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class TraverseNodesPropertiesTest extends TraversalServiceTest {
+ private TraversalRequest.Builder getTraversalRequestBuilder(SWHID src) {
+ return TraversalRequest.newBuilder().addSrc(src.toString());
+ }
+
+ private void checkHasAllFields(Message m) {
+ for (Descriptors.FieldDescriptor fd : m.getAllFields().keySet()) {
+ assertTrue(m.hasField(fd));
+ }
+ }
+
+ private void checkHasAllFieldsOfType(Node node) {
+ if (node.hasCnt()) {
+ checkHasAllFields(node.getCnt());
+ }
+ if (node.hasRev()) {
+ checkHasAllFields(node.getRev());
+ }
+ if (node.hasRel()) {
+ checkHasAllFields(node.getRel());
+ }
+ if (node.hasOri()) {
+ checkHasAllFields(node.getOri());
+ }
+ }
+
+ private void checkSuccessors(SwhUnidirectionalGraph g, Node node) {
+ HashMap<String, DirEntry[]> graphSuccessors = new HashMap<>();
+ ArcLabelledNodeIterator.LabelledArcIterator it = g.labelledSuccessors(g.getNodeId(new SWHID(node.getSwhid())));
+ long succ;
+ while ((succ = it.nextLong()) != -1) {
+ graphSuccessors.put(g.getSWHID(succ).toString(), (DirEntry[]) it.label().get());
+ }
+
+ assertEquals(node.getSuccessorList().stream().map(Successor::getSwhid).collect(Collectors.toSet()),
+ graphSuccessors.keySet());
+
+ for (Successor successor : node.getSuccessorList()) {
+ DirEntry[] expectedArray = graphSuccessors.get(successor.getSwhid());
+ HashMap<String, Integer> expectedLabels = new HashMap<>();
+ for (DirEntry dirEntry : expectedArray) {
+ expectedLabels.put(new String(g.getLabelName(dirEntry.filenameId)), dirEntry.permission);
+ }
+ for (EdgeLabel edgeLabel : successor.getLabelList()) {
+ assertTrue(expectedLabels.containsKey(edgeLabel.getName().toStringUtf8()));
+ if (edgeLabel.getPermission() > 0) {
+ assertEquals(edgeLabel.getPermission(), expectedLabels.get(edgeLabel.getName().toStringUtf8()));
+ }
+ }
+ }
+ }
+
+ @Test
+ public void forwardFromRoot() {
+ ArrayList<Node> response = new ArrayList<>();
+ client.traverse(getTraversalRequestBuilder(new SWHID(TEST_ORIGIN_ID)).build()).forEachRemaining(response::add);
+ for (Node node : response) {
+ checkHasAllFieldsOfType(node);
+ checkSuccessors(g.getForwardGraph(), node);
+ }
+ }
+
+ @Test
+ public void backwardFromLeaf() {
+ ArrayList<Node> response = new ArrayList<>();
+ client.traverse(getTraversalRequestBuilder(fakeSWHID("cnt", 4)).setDirection(GraphDirection.BACKWARD).build())
+ .forEachRemaining(response::add);
+ for (Node node : response) {
+ checkHasAllFieldsOfType(node);
+ checkSuccessors(g.getBackwardGraph(), node);
+ }
+ }
+
+ @Test
+ public void forwardFromRootMaskedLabels() {
+ ArrayList<Node> response = new ArrayList<>();
+ client.traverse(getTraversalRequestBuilder(new SWHID(TEST_ORIGIN_ID))
+ .setMask(FieldMask.newBuilder().addPaths("successor.swhid").addPaths("swhid").build()).build())
+ .forEachRemaining(response::add);
+ for (Node node : response) {
+ HashSet<String> graphSuccessors = new HashSet<>();
+ ArcLabelledNodeIterator.LabelledArcIterator it = g
+ .labelledSuccessors(g.getNodeId(new SWHID(node.getSwhid())));
+ long succ;
+ while ((succ = it.nextLong()) != -1) {
+ graphSuccessors.add(g.getSWHID(succ).toString());
+ }
+
+ assertEquals(node.getSuccessorList().stream().map(Successor::getSwhid).collect(Collectors.toSet()),
+ graphSuccessors);
+ }
+ }
+}
diff --git a/java/src/test/java/org/softwareheritage/graph/rpc/TraverseNodesTest.java b/java/src/test/java/org/softwareheritage/graph/rpc/TraverseNodesTest.java
new file mode 100644
--- /dev/null
+++ b/java/src/test/java/org/softwareheritage/graph/rpc/TraverseNodesTest.java
@@ -0,0 +1,250 @@
+package org.softwareheritage.graph.rpc;
+
+import io.grpc.Status;
+import io.grpc.StatusRuntimeException;
+import org.junit.jupiter.api.Test;
+import org.softwareheritage.graph.GraphTest;
+import org.softwareheritage.graph.SWHID;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class TraverseNodesTest extends TraversalServiceTest {
+ private TraversalRequest.Builder getTraversalRequestBuilder(SWHID src) {
+ return TraversalRequest.newBuilder().addSrc(src.toString());
+ }
+
+ @Test
+ public void testSrcErrors() {
+ StatusRuntimeException thrown;
+ thrown = assertThrows(StatusRuntimeException.class,
+ () -> client.traverse(TraversalRequest.newBuilder().addSrc(fakeSWHID("cnt", 404).toString()).build())
+ .forEachRemaining((n) -> {
+ }));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class,
+ () -> client
+ .traverse(TraversalRequest.newBuilder()
+ .addSrc("swh:1:lol:0000000000000000000000000000000000000001").build())
+ .forEachRemaining((n) -> {
+ }));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ thrown = assertThrows(StatusRuntimeException.class,
+ () -> client
+ .traverse(TraversalRequest.newBuilder()
+ .addSrc("swh:1:cnt:000000000000000000000000000000000000000z").build())
+ .forEachRemaining((n) -> {
+ }));
+ assertEquals(Status.INVALID_ARGUMENT.getCode(), thrown.getStatus().getCode());
+ }
+
+ @Test
+ public void forwardFromRoot() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.traverse(getTraversalRequestBuilder(new SWHID(TEST_ORIGIN_ID)).build()));
+ List<SWHID> expected = List.of(fakeSWHID("cnt", 1), fakeSWHID("cnt", 4), fakeSWHID("cnt", 5),
+ fakeSWHID("cnt", 7), fakeSWHID("dir", 2), fakeSWHID("dir", 6), fakeSWHID("dir", 8),
+ fakeSWHID("rel", 10), fakeSWHID("rev", 3), fakeSWHID("rev", 9), fakeSWHID("snp", 20),
+ new SWHID(TEST_ORIGIN_ID));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void forwardFromMiddle() {
+ ArrayList<SWHID> actual = getSWHIDs(client.traverse(getTraversalRequestBuilder(fakeSWHID("dir", 12)).build()));
+ List<SWHID> expected = List.of(fakeSWHID("cnt", 1), fakeSWHID("cnt", 4), fakeSWHID("cnt", 5),
+ fakeSWHID("cnt", 7), fakeSWHID("cnt", 11), fakeSWHID("dir", 6), fakeSWHID("dir", 8),
+ fakeSWHID("dir", 12));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void forwardRelRev() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.traverse(getTraversalRequestBuilder(fakeSWHID("rel", 10)).setEdges("rel:rev,rev:rev").build()));
+ List<SWHID> expected = List.of(fakeSWHID("rel", 10), fakeSWHID("rev", 9), fakeSWHID("rev", 3));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void forwardFilterReturnedNodesDir() {
+ ArrayList<SWHID> actual = getSWHIDs(client.traverse(getTraversalRequestBuilder(fakeSWHID("rel", 10))
+ .setReturnNodes(NodeFilter.newBuilder().setTypes("dir").build()).build()));
+ List<SWHID> expected = List.of(fakeSWHID("dir", 2), fakeSWHID("dir", 8), fakeSWHID("dir", 6));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void backwardFromRoot() {
+ ArrayList<SWHID> actual = getSWHIDs(client.traverse(
+ getTraversalRequestBuilder(new SWHID(TEST_ORIGIN_ID)).setDirection(GraphDirection.BACKWARD).build()));
+ List<SWHID> expected = List.of(new SWHID(TEST_ORIGIN_ID));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void backwardFromMiddle() {
+ ArrayList<SWHID> actual = getSWHIDs(client.traverse(
+ getTraversalRequestBuilder(fakeSWHID("dir", 12)).setDirection(GraphDirection.BACKWARD).build()));
+ List<SWHID> expected = List.of(fakeSWHID("dir", 12), fakeSWHID("rel", 19), fakeSWHID("rev", 13),
+ fakeSWHID("rev", 18));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void backwardFromLeaf() {
+ ArrayList<SWHID> actual = getSWHIDs(client.traverse(
+ getTraversalRequestBuilder(fakeSWHID("cnt", 4)).setDirection(GraphDirection.BACKWARD).build()));
+ List<SWHID> expected = List.of(new SWHID(TEST_ORIGIN_ID), fakeSWHID("cnt", 4), fakeSWHID("dir", 6),
+ fakeSWHID("dir", 8), fakeSWHID("dir", 12), fakeSWHID("rel", 10), fakeSWHID("rel", 19),
+ fakeSWHID("rev", 9), fakeSWHID("rev", 13), fakeSWHID("rev", 18), fakeSWHID("snp", 20));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void forwardSnpToRev() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.traverse(getTraversalRequestBuilder(fakeSWHID("snp", 20)).setEdges("snp:rev").build()));
+ List<SWHID> expected = List.of(fakeSWHID("rev", 9), fakeSWHID("snp", 20));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void forwardRelToRevRevToRev() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.traverse(getTraversalRequestBuilder(fakeSWHID("rel", 10)).setEdges("rel:rev,rev:rev").build()));
+ List<SWHID> expected = List.of(fakeSWHID("rel", 10), fakeSWHID("rev", 3), fakeSWHID("rev", 9));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void forwardRevToAllDirToAll() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.traverse(getTraversalRequestBuilder(fakeSWHID("rev", 13)).setEdges("rev:*,dir:*").build()));
+ List<SWHID> expected = List.of(fakeSWHID("cnt", 1), fakeSWHID("cnt", 4), fakeSWHID("cnt", 5),
+ fakeSWHID("cnt", 7), fakeSWHID("cnt", 11), fakeSWHID("dir", 2), fakeSWHID("dir", 6),
+ fakeSWHID("dir", 8), fakeSWHID("dir", 12), fakeSWHID("rev", 3), fakeSWHID("rev", 9),
+ fakeSWHID("rev", 13));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void forwardSnpToAllRevToAll() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.traverse(getTraversalRequestBuilder(fakeSWHID("snp", 20)).setEdges("snp:*,rev:*").build()));
+ List<SWHID> expected = List.of(fakeSWHID("dir", 2), fakeSWHID("dir", 8), fakeSWHID("rel", 10),
+ fakeSWHID("rev", 3), fakeSWHID("rev", 9), fakeSWHID("snp", 20));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void forwardNoEdges() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.traverse(getTraversalRequestBuilder(fakeSWHID("snp", 20)).setEdges("").build()));
+ List<SWHID> expected = List.of(fakeSWHID("snp", 20));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void backwardRevToRevRevToRel() {
+ ArrayList<SWHID> actual = getSWHIDs(client.traverse(getTraversalRequestBuilder(fakeSWHID("rev", 3))
+ .setEdges("rev:rev,rev:rel").setDirection(GraphDirection.BACKWARD).build()));
+ List<SWHID> expected = List.of(fakeSWHID("rel", 10), fakeSWHID("rel", 19), fakeSWHID("rev", 3),
+ fakeSWHID("rev", 9), fakeSWHID("rev", 13), fakeSWHID("rev", 18));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void forwardFromRootNodesOnly() {
+ ArrayList<SWHID> actual = getSWHIDs(
+ client.traverse(getTraversalRequestBuilder(new SWHID(TEST_ORIGIN_ID)).build()));
+ List<SWHID> expected = List.of(new SWHID(TEST_ORIGIN_ID), fakeSWHID("cnt", 1), fakeSWHID("cnt", 4),
+ fakeSWHID("cnt", 5), fakeSWHID("cnt", 7), fakeSWHID("dir", 2), fakeSWHID("dir", 6), fakeSWHID("dir", 8),
+ fakeSWHID("rel", 10), fakeSWHID("rev", 3), fakeSWHID("rev", 9), fakeSWHID("snp", 20));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void backwardRevToAllNodesOnly() {
+ ArrayList<SWHID> actual = getSWHIDs(client.traverse(getTraversalRequestBuilder(fakeSWHID("rev", 3))
+ .setDirection(GraphDirection.BACKWARD).setEdges("rev:*").build()));
+ List<SWHID> expected = List.of(fakeSWHID("rel", 10), fakeSWHID("rel", 19), fakeSWHID("rev", 3),
+ fakeSWHID("rev", 9), fakeSWHID("rev", 13), fakeSWHID("rev", 18), fakeSWHID("snp", 20));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void forwardMultipleSources() {
+ ArrayList<SWHID> actual = getSWHIDs(client.traverse(getTraversalRequestBuilder(fakeSWHID("snp", 20))
+ .addSrc(fakeSWHID("rel", 19).toString()).setMaxDepth(1).build()));
+ List<SWHID> expected = List.of(fakeSWHID("snp", 20), fakeSWHID("rel", 19), fakeSWHID("rel", 10),
+ fakeSWHID("rev", 9), fakeSWHID("rev", 18));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ @Test
+ public void backwardMultipleSources() {
+ ArrayList<SWHID> actual = getSWHIDs(client.traverse(getTraversalRequestBuilder(fakeSWHID("cnt", 5))
+ .addSrc(fakeSWHID("dir", 16).toString()).setMaxDepth(2).setDirection(GraphDirection.BACKWARD).build()));
+ List<SWHID> expected = List.of(fakeSWHID("cnt", 5), fakeSWHID("dir", 16), fakeSWHID("dir", 6),
+ fakeSWHID("dir", 8), fakeSWHID("dir", 17), fakeSWHID("rev", 18));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ // Go from rel 19 with various max depths
+ @Test
+ public void maxDepth() {
+ TraversalRequest.Builder builder = getTraversalRequestBuilder(fakeSWHID("rel", 19));
+
+ ArrayList<SWHID> actual;
+ List<SWHID> expected;
+
+ actual = getSWHIDs(client.traverse(builder.setMaxDepth(0).build()));
+ expected = List.of(fakeSWHID("rel", 19));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+
+ actual = getSWHIDs(client.traverse(builder.setMaxDepth(1).build()));
+ expected = List.of(fakeSWHID("rel", 19), fakeSWHID("rev", 18));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+
+ actual = getSWHIDs(client.traverse(builder.setMaxDepth(2).build()));
+ expected = List.of(fakeSWHID("rel", 19), fakeSWHID("rev", 18), fakeSWHID("rev", 13), fakeSWHID("dir", 17));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+
+ actual = getSWHIDs(client.traverse(builder.setMaxDepth(3).build()));
+ expected = List.of(fakeSWHID("rel", 19), fakeSWHID("rev", 18), fakeSWHID("rev", 13), fakeSWHID("dir", 17),
+ fakeSWHID("rev", 9), fakeSWHID("dir", 12), fakeSWHID("dir", 16), fakeSWHID("cnt", 14));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+
+ // Go from rel 19 with various max edges
+ @Test
+ public void maxEdges() {
+ TraversalRequest.Builder builder = getTraversalRequestBuilder(fakeSWHID("rel", 19));
+
+ ArrayList<SWHID> actual;
+ List<SWHID> expected;
+
+ actual = getSWHIDs(client.traverse(builder.setMaxEdges(1).build()));
+ expected = List.of(fakeSWHID("rel", 19));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+
+ actual = getSWHIDs(client.traverse(builder.setMaxEdges(3).build()));
+ expected = List.of(fakeSWHID("rel", 19), fakeSWHID("rev", 18));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+
+ actual = getSWHIDs(client.traverse(builder.setMaxEdges(7).build()));
+ expected = List.of(fakeSWHID("rel", 19), fakeSWHID("rev", 18), fakeSWHID("rev", 13), fakeSWHID("dir", 17),
+ fakeSWHID("cnt", 14));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+
+ actual = getSWHIDs(client.traverse(builder.setMaxEdges(12).build()));
+ expected = List.of(fakeSWHID("rel", 19), fakeSWHID("rev", 18), fakeSWHID("rev", 13), fakeSWHID("dir", 17),
+ fakeSWHID("rev", 9), fakeSWHID("dir", 12), fakeSWHID("dir", 16), fakeSWHID("cnt", 14),
+ fakeSWHID("cnt", 15));
+ GraphTest.assertEqualsAnyOrder(expected, actual);
+ }
+}
diff --git a/java/src/test/java/org/softwareheritage/graph/utils/ForkJoinBigQuickSort2Test.java b/java/src/test/java/org/softwareheritage/graph/utils/ForkJoinBigQuickSort2Test.java
--- a/java/src/test/java/org/softwareheritage/graph/utils/ForkJoinBigQuickSort2Test.java
+++ b/java/src/test/java/org/softwareheritage/graph/utils/ForkJoinBigQuickSort2Test.java
@@ -82,15 +82,5 @@
d[1][i] = random.nextInt();
sortBig2(d[0], d[1], 10, 100);
checkArraySorted(d[0], d[1], 10, 100);
-
- d[0] = new long[10000000];
- random = new Random(0);
- for (int i = d[0].length; i-- != 0;)
- d[0][i] = random.nextInt();
- d[1] = new long[d[0].length];
- for (int i = d[1].length; i-- != 0;)
- d[1][i] = random.nextInt();
- sortBig2(d[0], d[1]);
- checkArraySorted(d[0], d[1]);
}
}
diff --git a/java/src/test/java/org/softwareheritage/graph/utils/ForkJoinQuickSort3Test.java b/java/src/test/java/org/softwareheritage/graph/utils/ForkJoinQuickSort3Test.java
--- a/java/src/test/java/org/softwareheritage/graph/utils/ForkJoinQuickSort3Test.java
+++ b/java/src/test/java/org/softwareheritage/graph/utils/ForkJoinQuickSort3Test.java
@@ -86,18 +86,5 @@
d[2][i] = random.nextInt();
ForkJoinQuickSort3.parallelQuickSort(d[0], d[1], d[2], 10, 100);
checkArraySorted(d[0], d[1], d[2], 10, 100);
-
- d[0] = new long[10000000];
- random = new Random(0);
- for (int i = d[0].length; i-- != 0;)
- d[0][i] = random.nextInt();
- d[1] = new long[d[0].length];
- for (int i = d[1].length; i-- != 0;)
- d[1][i] = random.nextInt();
- d[2] = new long[d[0].length];
- for (int i = d[2].length; i-- != 0;)
- d[2][i] = random.nextInt();
- ForkJoinQuickSort3.parallelQuickSort(d[0], d[1], d[2]);
- checkArraySorted(d[0], d[1], d[2]);
}
}
diff --git a/mypy.ini b/mypy.ini
--- a/mypy.ini
+++ b/mypy.ini
@@ -1,6 +1,9 @@
[mypy]
namespace_packages = True
warn_unused_ignores = True
+exclude = (?x)(
+ ^swh/graph/rpc
+ )
# 3rd party libraries without stubs (yet)
diff --git a/proto/swhgraph.proto b/proto/swhgraph.proto
new file mode 100644
--- /dev/null
+++ b/proto/swhgraph.proto
@@ -0,0 +1,316 @@
+syntax = "proto3";
+
+import "google/protobuf/field_mask.proto";
+
+option java_multiple_files = true;
+option java_package = "org.softwareheritage.graph.rpc";
+option java_outer_classname = "GraphService";
+
+package swh.graph;
+
+/* Graph traversal service */
+service TraversalService {
+ /* GetNode returns a single Node and its properties. */
+ rpc GetNode (GetNodeRequest) returns (Node);
+
+ /* Traverse performs a breadth-first graph traversal from a set of source
+ * nodes, then streams the nodes it encounters (if they match a given
+ * return filter), along with their properties.
+ */
+ rpc Traverse (TraversalRequest) returns (stream Node);
+
+ /* FindPathTo searches for a shortest path between a set of source nodes
+ * and a node that matches a specific *criteria*.
+ *
+ * It does so by performing a breadth-first search from the source node,
+ * until any node that matches the given criteria is found, then follows
+ * back its parents to return a shortest path from the source set to that
+ * node.
+ */
+ rpc FindPathTo (FindPathToRequest) returns (Path);
+
+ /* FindPathBetween searches for a shortest path between a set of source
+ * nodes and a set of destination nodes.
+ *
+ * It does so by performing a *bidirectional breadth-first search*, i.e.,
+ * two parallel breadth-first searches, one from the source set ("src-BFS")
+ * and one from the destination set ("dst-BFS"), until both searches find a
+ * common node that joins their visited sets. This node is called the
+ * "midpoint node".
+ * The path returned is the path src -> ... -> midpoint -> ... -> dst,
+ * which is always a shortest path between src and dst.
+ *
+ * The graph direction of both BFS can be configured separately. By
+ * default, the dst-BFS will use the graph in the opposite direction than
+ * the src-BFS (if direction = FORWARD, by default direction_reverse =
+ * BACKWARD, and vice-versa). The default behavior is thus to search for
+ * a shortest path between two nodes in a given direction. However, one
+ * can also specify FORWARD or BACKWARD for *both* the src-BFS and the
+ * dst-BFS. This will search for a common descendant or a common ancestor
+ * between the two sets, respectively. These will be the midpoints of the
+ * returned path.
+ */
+ rpc FindPathBetween (FindPathBetweenRequest) returns (Path);
+
+ /* CountNodes does the same as Traverse, but only returns the number of
+ * nodes accessed during the traversal. */
+ rpc CountNodes (TraversalRequest) returns (CountResponse);
+
+ /* CountEdges does the same as Traverse, but only returns the number of
+ * edges accessed during the traversal. */
+ rpc CountEdges (TraversalRequest) returns (CountResponse);
+
+ /* Stats returns various statistics on the overall graph. */
+ rpc Stats (StatsRequest) returns (StatsResponse);
+}
+
+/* Direction of the graph */
+enum GraphDirection {
+ /* Forward DAG: ori -> snp -> rel -> rev -> dir -> cnt */
+ FORWARD = 0;
+ /* Transposed DAG: cnt -> dir -> rev -> rel -> snp -> ori */
+ BACKWARD = 1;
+}
+
+/* Describe a node to return */
+message GetNodeRequest {
+ /* SWHID of the node to return */
+ string swhid = 1;
+ /* FieldMask of which fields are to be returned (e.g., "swhid,cnt.length").
+ * By default, all fields are returned. */
+ optional google.protobuf.FieldMask mask = 8;
+}
+
+/* TraversalRequest describes how a breadth-first traversal should be
+ * performed, and what should be returned to the client. */
+message TraversalRequest {
+ /* Set of source nodes (SWHIDs) */
+ repeated string src = 1;
+ /* Direction of the graph to traverse. Defaults to FORWARD. */
+ GraphDirection direction = 2;
+ /* Edge restriction string (e.g. "rev:dir,dir:cnt").
+ * Defaults to "*" (all). */
+ optional string edges = 3;
+ /* Maximum number of edges accessed in the traversal, after which it stops.
+ * Defaults to infinite. */
+ optional int64 max_edges = 4;
+ /* Do not return nodes with a depth lower than this number.
+ * By default, all depths are returned. */
+ optional int64 min_depth = 5;
+ /* Maximum depth of the traversal, after which it stops.
+ * Defaults to infinite. */
+ optional int64 max_depth = 6;
+ /* Filter which nodes will be sent to the stream. By default, all nodes are
+ * returned. */
+ optional NodeFilter return_nodes = 7;
+ /* FieldMask of which fields are to be returned (e.g., "swhid,cnt.length").
+ * By default, all fields are returned. */
+ optional google.protobuf.FieldMask mask = 8;
+}
+
+/* FindPathToRequest describes a request to find a shortest path between a
+ * set of nodes and a given target criteria, as well as what should be returned
+ * in the path.
+ */
+message FindPathToRequest {
+ /* Set of source nodes (SWHIDs) */
+ repeated string src = 1;
+ /* Target criteria, i.e., what constitutes a valid path destination. */
+ NodeFilter target = 2;
+ /* Direction of the graph to traverse. Defaults to FORWARD. */
+ GraphDirection direction = 3;
+ /* Edge restriction string (e.g. "rev:dir,dir:cnt").
+ * Defaults to "*" (all). */
+ optional string edges = 4;
+ /* Maximum number of edges accessed in the traversal, after which it stops.
+ * Defaults to infinite. */
+ optional int64 max_edges = 5;
+ /* Maximum depth of the traversal, after which it stops.
+ * Defaults to infinite. */
+ optional int64 max_depth = 6;
+ /* FieldMask of which fields are to be returned (e.g., "swhid,cnt.length").
+ * By default, all fields are returned. */
+ optional google.protobuf.FieldMask mask = 7;
+}
+
+/* FindPathToRequest describes a request to find a shortest path between a
+ * set of source nodes and a set of destination nodes. It works by performing a
+ * bidirectional breadth-first traversal from both sets at the same time.
+ */
+message FindPathBetweenRequest {
+ /* Set of source nodes (SWHIDs) */
+ repeated string src = 1;
+ /* Set of destination nodes (SWHIDs) */
+ repeated string dst = 2;
+ /* Direction of the graph to traverse from the source set. Defaults to
+ * FORWARD. */
+ GraphDirection direction = 3;
+ /* Direction of the graph to traverse from the destination set. Defaults to
+ * the opposite of `direction`. If direction and direction_reverse are
+ * identical, it will find the first common successor of both sets in the
+ * given direction. */
+ optional GraphDirection direction_reverse = 4;
+ /* Edge restriction string for the traversal from the source set.
+ * (e.g. "rev:dir,dir:cnt"). Defaults to "*" (all). */
+ optional string edges = 5;
+ /* Edge restriction string for the reverse traversal from the destination
+ * set.
+ * If not specified:
+ * - If `edges` is not specified either, defaults to "*"
+ * - If direction == direction_reverse, defaults to `edges`
+ * - If direction != direction_reverse, defaults to the reverse of `edges`
+ * (e.g. "rev:dir" becomes "dir:rev").
+ */
+ optional string edges_reverse = 6;
+ /* Maximum number of edges accessed in the traversal, after which it stops.
+ * Defaults to infinite. */
+ optional int64 max_edges = 7;
+ /* Maximum depth of the traversal, after which it stops.
+ * Defaults to infinite. */
+ optional int64 max_depth = 8;
+ /* FieldMask of which fields are to be returned (e.g., "swhid,cnt.length").
+ * By default, all fields are returned. */
+ optional google.protobuf.FieldMask mask = 9;
+}
+
+/* Represents various criteria that make a given node "valid". A node is
+ * only valid if all the subcriteria present in this message are fulfilled.
+ */
+message NodeFilter {
+ /* Node restriction string. (e.g. "dir,cnt,rev"). Defaults to "*" (all). */
+ optional string types = 1;
+ /* Minimum number of successors encountered *during the traversal*.
+ * Default: no constraint */
+ optional int64 min_traversal_successors = 2;
+ /* Maximum number of successors encountered *during the traversal*.
+ * Default: no constraint */
+ optional int64 max_traversal_successors = 3;
+}
+
+/* Represents a node in the graph. */
+message Node {
+ /* The SWHID of the graph node. */
+ string swhid = 1;
+ /* List of relevant successors of this node. */
+ repeated Successor successor = 2;
+ /* Number of relevant successors. */
+ optional int64 num_successors = 9;
+ /* Node properties */
+ oneof data {
+ ContentData cnt = 3;
+ RevisionData rev = 5;
+ ReleaseData rel = 6;
+ OriginData ori = 8;
+ };
+}
+
+/* Represents a path in the graph. */
+message Path {
+ /* List of nodes in the path, from source to destination */
+ repeated Node node = 1;
+ /* Index of the "midpoint" of the path. For paths obtained with
+ * bidirectional search queries, this is the node that joined the two
+ * sets together. When looking for a common ancestor between two nodes by
+ * performing a FindPathBetween search with two backward graphs, this will
+ * be the index of the common ancestor in the path. */
+ optional int32 midpoint_index = 2;
+}
+
+/* Represents a successor of a given node. */
+message Successor {
+ /* The SWHID of the successor */
+ optional string swhid = 1;
+ /* A list of edge labels for the given edge */
+ repeated EdgeLabel label = 2;
+}
+
+/* Content node properties */
+message ContentData {
+ /* Length of the blob, in bytes */
+ optional int64 length = 1;
+ /* Whether the content was skipped during ingestion. */
+ optional bool is_skipped = 2;
+}
+
+/* Revision node properties */
+message RevisionData {
+ /* Revision author ID (anonymized) */
+ optional int64 author = 1;
+ /* UNIX timestamp of the revision date (UTC) */
+ optional int64 author_date = 2;
+ /* Timezone of the revision author date as an offset from UTC */
+ optional int32 author_date_offset = 3;
+ /* Revision committer ID (anonymized) */
+ optional int64 committer = 4;
+ /* UNIX timestamp of the revision committer date (UTC) */
+ optional int64 committer_date = 5;
+ /* Timezone of the revision committer date as an offset from UTC */
+ optional int32 committer_date_offset = 6;
+ /* Revision message */
+ optional bytes message = 7;
+}
+
+/* Release node properties */
+message ReleaseData {
+ /* Release author ID (anonymized) */
+ optional int64 author = 1;
+ /* UNIX timestamp of the release date (UTC) */
+ optional int64 author_date = 2;
+ /* Timezone of the release author date as an offset from UTC */
+ optional int32 author_date_offset = 3;
+ /* Release name */
+ optional bytes name = 4;
+ /* Release message */
+ optional bytes message = 5;
+}
+
+/* Origin node properties */
+message OriginData {
+ /* URL of the origin */
+ optional string url = 1;
+}
+
+message EdgeLabel {
+ /* Directory entry name for directories, branch name for snapshots */
+ bytes name = 1;
+ /* Entry permission (only set for directories). */
+ int32 permission = 2;
+}
+
+message CountResponse {
+ int64 count = 1;
+}
+
+message StatsRequest {
+}
+
+message StatsResponse {
+ /* Number of nodes in the graph */
+ int64 num_nodes = 1;
+ /* Number of edges in the graph */
+ int64 num_edges = 2;
+
+ /* Ratio between the graph size and the information-theoretical lower
+ * bound */
+ double compression_ratio = 3;
+ /* Number of bits per node (overall graph size in bits divided by the
+ * number of nodes) */
+ double bits_per_node = 4;
+ /* Number of bits per edge (overall graph size in bits divided by the
+ * number of arcs). */
+ double bits_per_edge = 5;
+ double avg_locality = 6;
+
+ /* Smallest indegree */
+ int64 indegree_min = 7;
+ /* Largest indegree */
+ int64 indegree_max = 8;
+ /* Average indegree */
+ double indegree_avg = 9;
+ /* Smallest outdegree */
+ int64 outdegree_min = 10;
+ /* Largest outdegree */
+ int64 outdegree_max = 11;
+ /* Average outdegree */
+ double outdegree_avg = 12;
+}
diff --git a/pyproject.toml b/pyproject.toml
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,5 +1,10 @@
[tool.black]
target-version = ['py37']
+extend-exclude = '''
+/(
+ | swh/graph/rpc
+)/
+'''
[tool.isort]
multi_line_output = 3
diff --git a/requirements-test.txt b/requirements-test.txt
--- a/requirements-test.txt
+++ b/requirements-test.txt
@@ -4,3 +4,5 @@
types-click
types-pyyaml
types-requests
+types-protobuf
+grpc-stubs
diff --git a/requirements.txt b/requirements.txt
--- a/requirements.txt
+++ b/requirements.txt
@@ -2,3 +2,5 @@
click
py4j
psutil
+grpcio-tools
+mypy-protobuf
diff --git a/setup.cfg b/setup.cfg
--- a/setup.cfg
+++ b/setup.cfg
@@ -6,3 +6,4 @@
select = C,E,F,W,B950
ignore = E203,E231,E501,W503
max-line-length = 88
+extend_exclude = swh/graph/rpc
diff --git a/swh/graph/backend.py b/swh/graph/backend.py
deleted file mode 100644
--- a/swh/graph/backend.py
+++ /dev/null
@@ -1,176 +0,0 @@
-# Copyright (C) 2019-2020 The Software Heritage developers
-# See the AUTHORS file at the top-level directory of this distribution
-# License: GNU General Public License version 3, or any later version
-# See top-level LICENSE file for more information
-
-import asyncio
-import contextlib
-import io
-import os
-import re
-import subprocess
-import sys
-import tempfile
-
-from py4j.java_gateway import JavaGateway
-from py4j.protocol import Py4JJavaError
-
-from swh.graph.config import check_config
-
-BUF_LINES = 1024
-
-
-def _get_pipe_stderr():
- # Get stderr if possible, or pipe to stdout if running with Jupyter.
- try:
- sys.stderr.fileno()
- except io.UnsupportedOperation:
- return subprocess.STDOUT
- else:
- return sys.stderr
-
-
-class Backend:
- def __init__(self, graph_path, config=None):
- self.gateway = None
- self.entry = None
- self.graph_path = graph_path
- self.config = check_config(config or {})
-
- def start_gateway(self):
- self.gateway = JavaGateway.launch_gateway(
- java_path=None,
- javaopts=self.config["java_tool_options"].split(),
- classpath=self.config["classpath"],
- die_on_exit=True,
- redirect_stdout=sys.stdout,
- redirect_stderr=_get_pipe_stderr(),
- )
- self.entry = self.gateway.jvm.org.softwareheritage.graph.Entry()
- self.entry.load_graph(self.graph_path)
- self.stream_proxy = JavaStreamProxy(self.entry)
-
- def stop_gateway(self):
- self.gateway.shutdown()
-
- def __enter__(self):
- self.start_gateway()
- return self
-
- def __exit__(self, exc_type, exc_value, tb):
- self.stop_gateway()
-
- def stats(self):
- return self.entry.stats()
-
- def check_swhid(self, swhid):
- try:
- self.entry.check_swhid(swhid)
- except Py4JJavaError as e:
- m = re.search(r"malformed SWHID: (\w+)", str(e))
- if m:
- raise ValueError(f"malformed SWHID: {m[1]}")
- m = re.search(r"Unknown SWHID: ([:\w]+)", str(e))
- if m:
- raise NameError(f"Unknown SWHID: {m[1]}")
- raise
-
- def count(self, ttype, *args):
- method = getattr(self.entry, "count_" + ttype)
- return method(*args)
-
- async def traversal(self, ttype, *args):
- method = getattr(self.stream_proxy, ttype)
- async for line in method(*args):
- yield line.decode().rstrip("\n")
-
-
-class JavaStreamProxy:
- """A proxy class for the org.softwareheritage.graph.Entry Java class that
- takes care of the setup and teardown of the named-pipe FIFO communication
- between Python and Java.
-
- Initialize JavaStreamProxy using:
-
- proxy = JavaStreamProxy(swh_entry_class_instance)
-
- Then you can call an Entry method and iterate on the FIFO results like
- this:
-
- async for value in proxy.java_method(arg1, arg2):
- print(value)
- """
-
- def __init__(self, entry):
- self.entry = entry
-
- async def read_node_ids(self, fname):
- loop = asyncio.get_event_loop()
- open_thread = loop.run_in_executor(None, open, fname, "rb")
-
- # Since the open() call on the FIFO is blocking until it is also opened
- # on the Java side, we await it with a timeout in case there is an
- # exception that prevents the write-side open().
- with (await asyncio.wait_for(open_thread, timeout=2)) as f:
-
- def read_n_lines(f, n):
- buf = []
- for _ in range(n):
- try:
- buf.append(next(f))
- except StopIteration:
- break
- return buf
-
- while True:
- lines = await loop.run_in_executor(None, read_n_lines, f, BUF_LINES)
- if not lines:
- break
- for line in lines:
- yield line
-
- class _HandlerWrapper:
- def __init__(self, handler):
- self._handler = handler
-
- def __getattr__(self, name):
- func = getattr(self._handler, name)
-
- async def java_call(*args, **kwargs):
- loop = asyncio.get_event_loop()
- await loop.run_in_executor(None, lambda: func(*args, **kwargs))
-
- def java_task(*args, **kwargs):
- return asyncio.create_task(java_call(*args, **kwargs))
-
- return java_task
-
- @contextlib.contextmanager
- def get_handler(self):
- with tempfile.TemporaryDirectory(prefix="swh-graph-") as tmpdirname:
- cli_fifo = os.path.join(tmpdirname, "swh-graph.fifo")
- os.mkfifo(cli_fifo)
- reader = self.read_node_ids(cli_fifo)
- query_handler = self.entry.get_handler(cli_fifo)
- handler = self._HandlerWrapper(query_handler)
- yield (handler, reader)
-
- def __getattr__(self, name):
- async def java_call_iterator(*args, **kwargs):
- with self.get_handler() as (handler, reader):
- java_task = getattr(handler, name)(*args, **kwargs)
- try:
- async for value in reader:
- yield value
- except asyncio.TimeoutError:
- # If the read-side open() timeouts, an exception on the
- # Java side probably happened that prevented the
- # write-side open(). We propagate this exception here if
- # that is the case.
- task_exc = java_task.exception()
- if task_exc:
- raise task_exc
- raise
- await java_task
-
- return java_call_iterator
diff --git a/swh/graph/cli.py b/swh/graph/cli.py
--- a/swh/graph/cli.py
+++ b/swh/graph/cli.py
@@ -122,9 +122,9 @@
@click.pass_context
def serve(ctx, host, port, graph):
"""run the graph RPC service"""
- import aiohttp
+ import aiohttp.web
- from swh.graph.server.app import make_app
+ from swh.graph.http_server import make_app
config = ctx.obj["config"]
config.setdefault("graph", {})
diff --git a/swh/graph/dot.py b/swh/graph/dot.py
deleted file mode 100644
--- a/swh/graph/dot.py
+++ /dev/null
@@ -1,68 +0,0 @@
-# Copyright (C) 2019 The Software Heritage developers
-# See the AUTHORS file at the top-level directory of this distribution
-# License: GNU General Public License version 3, or any later version
-# See top-level LICENSE file for more information
-
-import collections
-from functools import lru_cache
-import subprocess
-
-KIND_TO_SHAPE = {
- "ori": "egg",
- "snp": "doubleoctagon",
- "rel": "octagon",
- "rev": "diamond",
- "dir": "folder",
- "cnt": "oval",
-}
-
-
-@lru_cache()
-def dot_to_svg(dot):
- try:
- p = subprocess.run(
- ["dot", "-Tsvg"],
- input=dot,
- universal_newlines=True,
- capture_output=True,
- check=True,
- )
- except subprocess.CalledProcessError as e:
- raise RuntimeError(e.stderr) from e
- return p.stdout
-
-
-def graph_dot(nodes):
- ids = {n.id for n in nodes}
-
- by_kind = collections.defaultdict(list)
- for n in nodes:
- by_kind[n.kind].append(n)
-
- forward_edges = [
- (node.id, child.id)
- for node in nodes
- for child in node.children()
- if child.id in ids
- ]
- backward_edges = [
- (parent.id, node.id)
- for node in nodes
- for parent in node.parents()
- if parent.id in ids
- ]
- edges = set(forward_edges + backward_edges)
- edges_fmt = "\n".join("{} -> {};".format(a, b) for a, b in edges)
- nodes_fmt = "\n".join(node.dot_fragment() for node in nodes)
-
- s = """digraph G {{
- ranksep=1;
- nodesep=0.5;
-
- {nodes}
- {edges}
-
- }}""".format(
- nodes=nodes_fmt, edges=edges_fmt
- )
- return s
diff --git a/swh/graph/client.py b/swh/graph/http_client.py
rename from swh/graph/client.py
rename to swh/graph/http_client.py
diff --git a/swh/graph/naive_client.py b/swh/graph/http_naive_client.py
rename from swh/graph/naive_client.py
rename to swh/graph/http_naive_client.py
--- a/swh/graph/naive_client.py
+++ b/swh/graph/http_naive_client.py
@@ -22,7 +22,7 @@
from swh.model.swhids import CoreSWHID, ExtendedSWHID, ValidationError
-from .client import GraphArgumentException
+from .http_client import GraphArgumentException
_NODE_TYPES = "ori|snp|rel|rev|dir|cnt"
NODES_RE = re.compile(rf"(\*|{_NODE_TYPES})")
@@ -81,10 +81,10 @@
class NaiveClient:
- """An alternative implementation of :class:`swh.graph.backend.Backend`,
- written in pure-python and meant for simulating it in other components' test
- cases; constructed from a list of nodes and (directed) edges, both
- represented as SWHIDs.
+ """An alternative implementation of the graph server, written in
+ pure-python and meant for simulating it in other components' test cases;
+ constructed from a list of nodes and (directed) edges, both represented as
+ SWHIDs.
It is NOT meant to be efficient in any way; only to be a very simple
implementation that provides the same behavior.
@@ -124,26 +124,22 @@
def stats(self) -> Dict:
return {
- "counts": {
- "nodes": len(self.graph.nodes),
- "edges": sum(map(len, self.graph.forward_edges.values())),
- },
- "ratios": {
- "compression": 1.0,
- "bits_per_edge": 100.0,
- "bits_per_node": 100.0,
- "avg_locality": 0.0,
- },
- "indegree": {
- "min": min(map(len, self.graph.backward_edges.values())),
- "max": max(map(len, self.graph.backward_edges.values())),
- "avg": statistics.mean(map(len, self.graph.backward_edges.values())),
- },
- "outdegree": {
- "min": min(map(len, self.graph.forward_edges.values())),
- "max": max(map(len, self.graph.forward_edges.values())),
- "avg": statistics.mean(map(len, self.graph.forward_edges.values())),
- },
+ "num_nodes": len(self.graph.nodes),
+ "num_edges": sum(map(len, self.graph.forward_edges.values())),
+ "compression_ratio": 1.0,
+ "bits_per_edge": 100.0,
+ "bits_per_node": 100.0,
+ "avg_locality": 0.0,
+ "indegree_min": min(map(len, self.graph.backward_edges.values())),
+ "indegree_max": max(map(len, self.graph.backward_edges.values())),
+ "indegree_avg": statistics.mean(
+ map(len, self.graph.backward_edges.values())
+ ),
+ "outdegree_min": min(map(len, self.graph.forward_edges.values())),
+ "outdegree_max": max(map(len, self.graph.forward_edges.values())),
+ "outdegree_avg": statistics.mean(
+ map(len, self.graph.forward_edges.values())
+ ),
}
@check_arguments
diff --git a/swh/graph/server/app.py b/swh/graph/http_server.py
rename from swh/graph/server/app.py
rename to swh/graph/http_server.py
--- a/swh/graph/server/app.py
+++ b/swh/graph/http_server.py
@@ -8,16 +8,26 @@
FIFO as a transport to stream integers between the two languages.
"""
-import asyncio
-from collections import deque
+import json
import os
from typing import Optional
+import aiohttp.test_utils
import aiohttp.web
+from google.protobuf import json_format
+from google.protobuf.field_mask_pb2 import FieldMask
+import grpc
from swh.core.api.asynchronous import RPCServerApp
from swh.core.config import read as config_read
-from swh.graph.backend import Backend
+from swh.graph.rpc.swhgraph_pb2 import (
+ GetNodeRequest,
+ NodeFilter,
+ StatsRequest,
+ TraversalRequest,
+)
+from swh.graph.rpc.swhgraph_pb2_grpc import TraversalServiceStub
+from swh.graph.rpc_server import spawn_java_rpc_server
from swh.model.swhids import EXTENDED_SWHID_TYPES
try:
@@ -34,18 +44,21 @@
class GraphServerApp(RPCServerApp):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
- self.on_startup.append(self._start_gateway)
- self.on_shutdown.append(self._stop_gateway)
+ self.on_startup.append(self._start)
+ self.on_shutdown.append(self._stop)
@staticmethod
- async def _start_gateway(app):
- # Equivalent to entering `with app["backend"]:`
- app["backend"].start_gateway()
+ async def _start(app):
+ app["channel"] = grpc.aio.insecure_channel(app["rpc_url"])
+ await app["channel"].__aenter__()
+ app["rpc_client"] = TraversalServiceStub(app["channel"])
+ await app["rpc_client"].Stats(StatsRequest(), wait_for_ready=True)
@staticmethod
- async def _stop_gateway(app):
- # Equivalent to exiting `with app["backend"]:` with no error
- app["backend"].stop_gateway()
+ async def _stop(app):
+ await app["channel"].__aexit__(None, None, None)
+ if app.get("local_server"):
+ app["local_server"].terminate()
async def index(request):
@@ -70,14 +83,14 @@
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
- self.backend = self.request.app["backend"]
+ self.rpc_client: TraversalServiceStub = self.request.app["rpc_client"]
def get_direction(self):
"""Validate HTTP query parameter `direction`"""
s = self.request.query.get("direction", "forward")
if s not in ("forward", "backward"):
raise aiohttp.web.HTTPBadRequest(text=f"invalid direction: {s}")
- return s
+ return s.upper()
def get_edges(self):
"""Validate HTTP query parameter `edges`, i.e., edge restrictions"""
@@ -134,12 +147,15 @@
except ValueError:
raise aiohttp.web.HTTPBadRequest(text=f"invalid max_edges value: {s}")
- def check_swhid(self, swhid):
+ async def check_swhid(self, swhid):
"""Validate that the given SWHID exists in the graph"""
try:
- self.backend.check_swhid(swhid)
- except (NameError, ValueError) as e:
- raise aiohttp.web.HTTPBadRequest(text=str(e))
+ await self.rpc_client.GetNode(
+ GetNodeRequest(swhid=swhid, mask=FieldMask(paths=["swhid"]))
+ )
+ except grpc.aio.AioRpcError as e:
+ if e.code() == grpc.StatusCode.INVALID_ARGUMENT:
+ raise aiohttp.web.HTTPBadRequest(text=str(e.details()))
class StreamingGraphView(GraphView):
@@ -193,109 +209,70 @@
"""View showing some statistics on the graph"""
async def get(self):
- stats = self.backend.stats()
- return aiohttp.web.Response(body=stats, content_type="application/json")
+ res = await self.rpc_client.Stats(StatsRequest())
+ stats = json_format.MessageToDict(
+ res, including_default_value_fields=True, preserving_proto_field_name=True
+ )
+ # Int64 fields are serialized as strings by default.
+ for descriptor in res.DESCRIPTOR.fields:
+ if descriptor.type == descriptor.TYPE_INT64:
+ try:
+ stats[descriptor.name] = int(stats[descriptor.name])
+ except KeyError:
+ pass
+ json_body = json.dumps(stats, indent=4, sort_keys=True)
+ return aiohttp.web.Response(body=json_body, content_type="application/json")
class SimpleTraversalView(StreamingGraphView):
"""Base class for views of simple traversals"""
- simple_traversal_type: Optional[str] = None
-
async def prepare_response(self):
- self.src = self.request.match_info["src"]
- self.edges = self.get_edges()
- self.direction = self.get_direction()
- self.max_edges = self.get_max_edges()
- self.return_types = self.get_return_types()
- self.check_swhid(self.src)
+ src = self.request.match_info["src"]
+ self.traversal_request = TraversalRequest(
+ src=[src],
+ edges=self.get_edges(),
+ direction=self.get_direction(),
+ return_nodes=NodeFilter(types=self.get_return_types()),
+ mask=FieldMask(paths=["swhid"]),
+ )
+ if self.get_max_edges():
+ self.traversal_request.max_edges = self.get_max_edges()
+ await self.check_swhid(src)
+ self.configure_request()
+
+ def configure_request(self):
+ pass
async def stream_response(self):
- async for res_line in self.backend.traversal(
- self.simple_traversal_type,
- self.direction,
- self.edges,
- self.src,
- self.max_edges,
- self.return_types,
- ):
- await self.stream_line(res_line)
+ async for node in self.rpc_client.Traverse(self.traversal_request):
+ await self.stream_line(node.swhid)
class LeavesView(SimpleTraversalView):
- simple_traversal_type = "leaves"
+ def configure_request(self):
+ self.traversal_request.return_nodes.max_traversal_successors = 0
class NeighborsView(SimpleTraversalView):
- simple_traversal_type = "neighbors"
+ def configure_request(self):
+ self.traversal_request.min_depth = 1
+ self.traversal_request.max_depth = 1
class VisitNodesView(SimpleTraversalView):
- simple_traversal_type = "visit_nodes"
+ pass
class VisitEdgesView(SimpleTraversalView):
- simple_traversal_type = "visit_edges"
-
-
-class WalkView(StreamingGraphView):
- async def prepare_response(self):
- self.src = self.request.match_info["src"]
- self.dst = self.request.match_info["dst"]
-
- self.edges = self.get_edges()
- self.direction = self.get_direction()
- self.algo = self.get_traversal()
- self.limit = self.get_limit()
- self.max_edges = self.get_max_edges()
- self.return_types = self.get_return_types()
-
- self.check_swhid(self.src)
- if self.dst not in EXTENDED_SWHID_TYPES:
- self.check_swhid(self.dst)
-
- async def get_walk_iterator(self):
- return self.backend.traversal(
- "walk",
- self.direction,
- self.edges,
- self.algo,
- self.src,
- self.dst,
- self.max_edges,
- self.return_types,
- )
+ def configure_request(self):
+ self.traversal_request.mask.paths.extend(["successor", "successor.swhid"])
+ # self.traversal_request.return_fields.successor = True
async def stream_response(self):
- it = self.get_walk_iterator()
- if self.limit < 0:
- queue = deque(maxlen=-self.limit)
- async for res_swhid in it:
- queue.append(res_swhid)
- while queue:
- await self.stream_line(queue.popleft())
- else:
- count = 0
- async for res_swhid in it:
- if self.limit == 0 or count < self.limit:
- await self.stream_line(res_swhid)
- count += 1
- else:
- break
-
-
-class RandomWalkView(WalkView):
- def get_walk_iterator(self):
- return self.backend.traversal(
- "random_walk",
- self.direction,
- self.edges,
- RANDOM_RETRIES,
- self.src,
- self.dst,
- self.max_edges,
- self.return_types,
- )
+ async for node in self.rpc_client.Traverse(self.traversal_request):
+ for succ in node.successor:
+ await self.stream_line(node.swhid + " " + succ.swhid)
class CountView(GraphView):
@@ -304,44 +281,48 @@
count_type: Optional[str] = None
async def get(self):
- self.src = self.request.match_info["src"]
- self.check_swhid(self.src)
-
- self.edges = self.get_edges()
- self.direction = self.get_direction()
- self.max_edges = self.get_max_edges()
-
- loop = asyncio.get_event_loop()
- cnt = await loop.run_in_executor(
- None,
- self.backend.count,
- self.count_type,
- self.direction,
- self.edges,
- self.src,
- self.max_edges,
+ src = self.request.match_info["src"]
+ self.traversal_request = TraversalRequest(
+ src=[src],
+ edges=self.get_edges(),
+ direction=self.get_direction(),
+ return_nodes=NodeFilter(types=self.get_return_types()),
+ mask=FieldMask(paths=["swhid"]),
+ )
+ if self.get_max_edges():
+ self.traversal_request.max_edges = self.get_max_edges()
+ self.configure_request()
+ res = await self.rpc_client.CountNodes(self.traversal_request)
+ return aiohttp.web.Response(
+ body=str(res.count), content_type="application/json"
)
- return aiohttp.web.Response(body=str(cnt), content_type="application/json")
+
+ def configure_request(self):
+ pass
class CountNeighborsView(CountView):
- count_type = "neighbors"
+ def configure_request(self):
+ self.traversal_request.min_depth = 1
+ self.traversal_request.max_depth = 1
class CountLeavesView(CountView):
- count_type = "leaves"
+ def configure_request(self):
+ self.traversal_request.return_nodes.max_traversal_successors = 0
class CountVisitNodesView(CountView):
- count_type = "visit_nodes"
+ pass
-def make_app(config=None, backend=None, **kwargs):
- if (config is None) == (backend is None):
- raise ValueError("make_app() expects exactly one of 'config' or 'backend'")
- if backend is None:
- backend = Backend(graph_path=config["graph"]["path"], config=config["graph"])
+def make_app(config=None, rpc_url=None, **kwargs):
app = GraphServerApp(**kwargs)
+
+ if rpc_url is None:
+ app["local_server"], port = spawn_java_rpc_server(config)
+ rpc_url = f"localhost:{port}"
+
app.add_routes(
[
aiohttp.web.get("/", index),
@@ -351,16 +332,13 @@
aiohttp.web.view("/graph/neighbors/{src}", NeighborsView),
aiohttp.web.view("/graph/visit/nodes/{src}", VisitNodesView),
aiohttp.web.view("/graph/visit/edges/{src}", VisitEdgesView),
- # temporarily disabled in wait of a proper fix for T1969
- # aiohttp.web.view("/graph/walk/{src}/{dst}", WalkView)
- aiohttp.web.view("/graph/randomwalk/{src}/{dst}", RandomWalkView),
aiohttp.web.view("/graph/neighbors/count/{src}", CountNeighborsView),
aiohttp.web.view("/graph/leaves/count/{src}", CountLeavesView),
aiohttp.web.view("/graph/visit/nodes/count/{src}", CountVisitNodesView),
]
)
- app["backend"] = backend
+ app["rpc_url"] = rpc_url
return app
diff --git a/swh/graph/rpc/swhgraph.proto b/swh/graph/rpc/swhgraph.proto
new file mode 120000
--- /dev/null
+++ b/swh/graph/rpc/swhgraph.proto
@@ -0,0 +1 @@
+../../../proto/swhgraph.proto
\ No newline at end of file
diff --git a/swh/graph/rpc/swhgraph_pb2.py b/swh/graph/rpc/swhgraph_pb2.py
new file mode 100644
--- /dev/null
+++ b/swh/graph/rpc/swhgraph_pb2.py
@@ -0,0 +1,196 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler. DO NOT EDIT!
+# source: swh/graph/rpc/swhgraph.proto
+"""Generated protocol buffer code."""
+from google.protobuf.internal import enum_type_wrapper
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import descriptor_pool as _descriptor_pool
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.protobuf import field_mask_pb2 as google_dot_protobuf_dot_field__mask__pb2
+
+
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1cswh/graph/rpc/swhgraph.proto\x12\tswh.graph\x1a google/protobuf/field_mask.proto\"W\n\x0eGetNodeRequest\x12\r\n\x05swhid\x18\x01 \x01(\t\x12-\n\x04mask\x18\x08 \x01(\x0b\x32\x1a.google.protobuf.FieldMaskH\x00\x88\x01\x01\x42\x07\n\x05_mask\"\xd8\x02\n\x10TraversalRequest\x12\x0b\n\x03src\x18\x01 \x03(\t\x12,\n\tdirection\x18\x02 \x01(\x0e\x32\x19.swh.graph.GraphDirection\x12\x12\n\x05\x65\x64ges\x18\x03 \x01(\tH\x00\x88\x01\x01\x12\x16\n\tmax_edges\x18\x04 \x01(\x03H\x01\x88\x01\x01\x12\x16\n\tmin_depth\x18\x05 \x01(\x03H\x02\x88\x01\x01\x12\x16\n\tmax_depth\x18\x06 \x01(\x03H\x03\x88\x01\x01\x12\x30\n\x0creturn_nodes\x18\x07 \x01(\x0b\x32\x15.swh.graph.NodeFilterH\x04\x88\x01\x01\x12-\n\x04mask\x18\x08 \x01(\x0b\x32\x1a.google.protobuf.FieldMaskH\x05\x88\x01\x01\x42\x08\n\x06_edgesB\x0c\n\n_max_edgesB\x0c\n\n_min_depthB\x0c\n\n_max_depthB\x0f\n\r_return_nodesB\x07\n\x05_mask\"\x97\x02\n\x11\x46indPathToRequest\x12\x0b\n\x03src\x18\x01 \x03(\t\x12%\n\x06target\x18\x02 \x01(\x0b\x32\x15.swh.graph.NodeFilter\x12,\n\tdirection\x18\x03 \x01(\x0e\x32\x19.swh.graph.GraphDirection\x12\x12\n\x05\x65\x64ges\x18\x04 \x01(\tH\x00\x88\x01\x01\x12\x16\n\tmax_edges\x18\x05 \x01(\x03H\x01\x88\x01\x01\x12\x16\n\tmax_depth\x18\x06 \x01(\x03H\x02\x88\x01\x01\x12-\n\x04mask\x18\x07 \x01(\x0b\x32\x1a.google.protobuf.FieldMaskH\x03\x88\x01\x01\x42\x08\n\x06_edgesB\x0c\n\n_max_edgesB\x0c\n\n_max_depthB\x07\n\x05_mask\"\x81\x03\n\x16\x46indPathBetweenRequest\x12\x0b\n\x03src\x18\x01 \x03(\t\x12\x0b\n\x03\x64st\x18\x02 \x03(\t\x12,\n\tdirection\x18\x03 \x01(\x0e\x32\x19.swh.graph.GraphDirection\x12\x39\n\x11\x64irection_reverse\x18\x04 \x01(\x0e\x32\x19.swh.graph.GraphDirectionH\x00\x88\x01\x01\x12\x12\n\x05\x65\x64ges\x18\x05 \x01(\tH\x01\x88\x01\x01\x12\x1a\n\redges_reverse\x18\x06 \x01(\tH\x02\x88\x01\x01\x12\x16\n\tmax_edges\x18\x07 \x01(\x03H\x03\x88\x01\x01\x12\x16\n\tmax_depth\x18\x08 \x01(\x03H\x04\x88\x01\x01\x12-\n\x04mask\x18\t \x01(\x0b\x32\x1a.google.protobuf.FieldMaskH\x05\x88\x01\x01\x42\x14\n\x12_direction_reverseB\x08\n\x06_edgesB\x10\n\x0e_edges_reverseB\x0c\n\n_max_edgesB\x0c\n\n_max_depthB\x07\n\x05_mask\"\xb2\x01\n\nNodeFilter\x12\x12\n\x05types\x18\x01 \x01(\tH\x00\x88\x01\x01\x12%\n\x18min_traversal_successors\x18\x02 \x01(\x03H\x01\x88\x01\x01\x12%\n\x18max_traversal_successors\x18\x03 \x01(\x03H\x02\x88\x01\x01\x42\x08\n\x06_typesB\x1b\n\x19_min_traversal_successorsB\x1b\n\x19_max_traversal_successors\"\x92\x02\n\x04Node\x12\r\n\x05swhid\x18\x01 \x01(\t\x12\'\n\tsuccessor\x18\x02 \x03(\x0b\x32\x14.swh.graph.Successor\x12\x1b\n\x0enum_successors\x18\t \x01(\x03H\x01\x88\x01\x01\x12%\n\x03\x63nt\x18\x03 \x01(\x0b\x32\x16.swh.graph.ContentDataH\x00\x12&\n\x03rev\x18\x05 \x01(\x0b\x32\x17.swh.graph.RevisionDataH\x00\x12%\n\x03rel\x18\x06 \x01(\x0b\x32\x16.swh.graph.ReleaseDataH\x00\x12$\n\x03ori\x18\x08 \x01(\x0b\x32\x15.swh.graph.OriginDataH\x00\x42\x06\n\x04\x64\x61taB\x11\n\x0f_num_successors\"U\n\x04Path\x12\x1d\n\x04node\x18\x01 \x03(\x0b\x32\x0f.swh.graph.Node\x12\x1b\n\x0emidpoint_index\x18\x02 \x01(\x05H\x00\x88\x01\x01\x42\x11\n\x0f_midpoint_index\"N\n\tSuccessor\x12\x12\n\x05swhid\x18\x01 \x01(\tH\x00\x88\x01\x01\x12#\n\x05label\x18\x02 \x03(\x0b\x32\x14.swh.graph.EdgeLabelB\x08\n\x06_swhid\"U\n\x0b\x43ontentData\x12\x13\n\x06length\x18\x01 \x01(\x03H\x00\x88\x01\x01\x12\x17\n\nis_skipped\x18\x02 \x01(\x08H\x01\x88\x01\x01\x42\t\n\x07_lengthB\r\n\x0b_is_skipped\"\xc6\x02\n\x0cRevisionData\x12\x13\n\x06\x61uthor\x18\x01 \x01(\x03H\x00\x88\x01\x01\x12\x18\n\x0b\x61uthor_date\x18\x02 \x01(\x03H\x01\x88\x01\x01\x12\x1f\n\x12\x61uthor_date_offset\x18\x03 \x01(\x05H\x02\x88\x01\x01\x12\x16\n\tcommitter\x18\x04 \x01(\x03H\x03\x88\x01\x01\x12\x1b\n\x0e\x63ommitter_date\x18\x05 \x01(\x03H\x04\x88\x01\x01\x12\"\n\x15\x63ommitter_date_offset\x18\x06 \x01(\x05H\x05\x88\x01\x01\x12\x14\n\x07message\x18\x07 \x01(\x0cH\x06\x88\x01\x01\x42\t\n\x07_authorB\x0e\n\x0c_author_dateB\x15\n\x13_author_date_offsetB\x0c\n\n_committerB\x11\n\x0f_committer_dateB\x18\n\x16_committer_date_offsetB\n\n\x08_message\"\xcd\x01\n\x0bReleaseData\x12\x13\n\x06\x61uthor\x18\x01 \x01(\x03H\x00\x88\x01\x01\x12\x18\n\x0b\x61uthor_date\x18\x02 \x01(\x03H\x01\x88\x01\x01\x12\x1f\n\x12\x61uthor_date_offset\x18\x03 \x01(\x05H\x02\x88\x01\x01\x12\x11\n\x04name\x18\x04 \x01(\x0cH\x03\x88\x01\x01\x12\x14\n\x07message\x18\x05 \x01(\x0cH\x04\x88\x01\x01\x42\t\n\x07_authorB\x0e\n\x0c_author_dateB\x15\n\x13_author_date_offsetB\x07\n\x05_nameB\n\n\x08_message\"&\n\nOriginData\x12\x10\n\x03url\x18\x01 \x01(\tH\x00\x88\x01\x01\x42\x06\n\x04_url\"-\n\tEdgeLabel\x12\x0c\n\x04name\x18\x01 \x01(\x0c\x12\x12\n\npermission\x18\x02 \x01(\x05\"\x1e\n\rCountResponse\x12\r\n\x05\x63ount\x18\x01 \x01(\x03\"\x0e\n\x0cStatsRequest\"\x9b\x02\n\rStatsResponse\x12\x11\n\tnum_nodes\x18\x01 \x01(\x03\x12\x11\n\tnum_edges\x18\x02 \x01(\x03\x12\x19\n\x11\x63ompression_ratio\x18\x03 \x01(\x01\x12\x15\n\rbits_per_node\x18\x04 \x01(\x01\x12\x15\n\rbits_per_edge\x18\x05 \x01(\x01\x12\x14\n\x0c\x61vg_locality\x18\x06 \x01(\x01\x12\x14\n\x0cindegree_min\x18\x07 \x01(\x03\x12\x14\n\x0cindegree_max\x18\x08 \x01(\x03\x12\x14\n\x0cindegree_avg\x18\t \x01(\x01\x12\x15\n\routdegree_min\x18\n \x01(\x03\x12\x15\n\routdegree_max\x18\x0b \x01(\x03\x12\x15\n\routdegree_avg\x18\x0c \x01(\x01*+\n\x0eGraphDirection\x12\x0b\n\x07\x46ORWARD\x10\x00\x12\x0c\n\x08\x42\x41\x43KWARD\x10\x01\x32\xcf\x03\n\x10TraversalService\x12\x35\n\x07GetNode\x12\x19.swh.graph.GetNodeRequest\x1a\x0f.swh.graph.Node\x12:\n\x08Traverse\x12\x1b.swh.graph.TraversalRequest\x1a\x0f.swh.graph.Node0\x01\x12;\n\nFindPathTo\x12\x1c.swh.graph.FindPathToRequest\x1a\x0f.swh.graph.Path\x12\x45\n\x0f\x46indPathBetween\x12!.swh.graph.FindPathBetweenRequest\x1a\x0f.swh.graph.Path\x12\x43\n\nCountNodes\x12\x1b.swh.graph.TraversalRequest\x1a\x18.swh.graph.CountResponse\x12\x43\n\nCountEdges\x12\x1b.swh.graph.TraversalRequest\x1a\x18.swh.graph.CountResponse\x12:\n\x05Stats\x12\x17.swh.graph.StatsRequest\x1a\x18.swh.graph.StatsResponseB0\n\x1eorg.softwareheritage.graph.rpcB\x0cGraphServiceP\x01\x62\x06proto3')
+
+_GRAPHDIRECTION = DESCRIPTOR.enum_types_by_name['GraphDirection']
+GraphDirection = enum_type_wrapper.EnumTypeWrapper(_GRAPHDIRECTION)
+FORWARD = 0
+BACKWARD = 1
+
+
+_GETNODEREQUEST = DESCRIPTOR.message_types_by_name['GetNodeRequest']
+_TRAVERSALREQUEST = DESCRIPTOR.message_types_by_name['TraversalRequest']
+_FINDPATHTOREQUEST = DESCRIPTOR.message_types_by_name['FindPathToRequest']
+_FINDPATHBETWEENREQUEST = DESCRIPTOR.message_types_by_name['FindPathBetweenRequest']
+_NODEFILTER = DESCRIPTOR.message_types_by_name['NodeFilter']
+_NODE = DESCRIPTOR.message_types_by_name['Node']
+_PATH = DESCRIPTOR.message_types_by_name['Path']
+_SUCCESSOR = DESCRIPTOR.message_types_by_name['Successor']
+_CONTENTDATA = DESCRIPTOR.message_types_by_name['ContentData']
+_REVISIONDATA = DESCRIPTOR.message_types_by_name['RevisionData']
+_RELEASEDATA = DESCRIPTOR.message_types_by_name['ReleaseData']
+_ORIGINDATA = DESCRIPTOR.message_types_by_name['OriginData']
+_EDGELABEL = DESCRIPTOR.message_types_by_name['EdgeLabel']
+_COUNTRESPONSE = DESCRIPTOR.message_types_by_name['CountResponse']
+_STATSREQUEST = DESCRIPTOR.message_types_by_name['StatsRequest']
+_STATSRESPONSE = DESCRIPTOR.message_types_by_name['StatsResponse']
+GetNodeRequest = _reflection.GeneratedProtocolMessageType('GetNodeRequest', (_message.Message,), {
+ 'DESCRIPTOR' : _GETNODEREQUEST,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.GetNodeRequest)
+ })
+_sym_db.RegisterMessage(GetNodeRequest)
+
+TraversalRequest = _reflection.GeneratedProtocolMessageType('TraversalRequest', (_message.Message,), {
+ 'DESCRIPTOR' : _TRAVERSALREQUEST,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.TraversalRequest)
+ })
+_sym_db.RegisterMessage(TraversalRequest)
+
+FindPathToRequest = _reflection.GeneratedProtocolMessageType('FindPathToRequest', (_message.Message,), {
+ 'DESCRIPTOR' : _FINDPATHTOREQUEST,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.FindPathToRequest)
+ })
+_sym_db.RegisterMessage(FindPathToRequest)
+
+FindPathBetweenRequest = _reflection.GeneratedProtocolMessageType('FindPathBetweenRequest', (_message.Message,), {
+ 'DESCRIPTOR' : _FINDPATHBETWEENREQUEST,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.FindPathBetweenRequest)
+ })
+_sym_db.RegisterMessage(FindPathBetweenRequest)
+
+NodeFilter = _reflection.GeneratedProtocolMessageType('NodeFilter', (_message.Message,), {
+ 'DESCRIPTOR' : _NODEFILTER,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.NodeFilter)
+ })
+_sym_db.RegisterMessage(NodeFilter)
+
+Node = _reflection.GeneratedProtocolMessageType('Node', (_message.Message,), {
+ 'DESCRIPTOR' : _NODE,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.Node)
+ })
+_sym_db.RegisterMessage(Node)
+
+Path = _reflection.GeneratedProtocolMessageType('Path', (_message.Message,), {
+ 'DESCRIPTOR' : _PATH,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.Path)
+ })
+_sym_db.RegisterMessage(Path)
+
+Successor = _reflection.GeneratedProtocolMessageType('Successor', (_message.Message,), {
+ 'DESCRIPTOR' : _SUCCESSOR,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.Successor)
+ })
+_sym_db.RegisterMessage(Successor)
+
+ContentData = _reflection.GeneratedProtocolMessageType('ContentData', (_message.Message,), {
+ 'DESCRIPTOR' : _CONTENTDATA,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.ContentData)
+ })
+_sym_db.RegisterMessage(ContentData)
+
+RevisionData = _reflection.GeneratedProtocolMessageType('RevisionData', (_message.Message,), {
+ 'DESCRIPTOR' : _REVISIONDATA,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.RevisionData)
+ })
+_sym_db.RegisterMessage(RevisionData)
+
+ReleaseData = _reflection.GeneratedProtocolMessageType('ReleaseData', (_message.Message,), {
+ 'DESCRIPTOR' : _RELEASEDATA,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.ReleaseData)
+ })
+_sym_db.RegisterMessage(ReleaseData)
+
+OriginData = _reflection.GeneratedProtocolMessageType('OriginData', (_message.Message,), {
+ 'DESCRIPTOR' : _ORIGINDATA,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.OriginData)
+ })
+_sym_db.RegisterMessage(OriginData)
+
+EdgeLabel = _reflection.GeneratedProtocolMessageType('EdgeLabel', (_message.Message,), {
+ 'DESCRIPTOR' : _EDGELABEL,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.EdgeLabel)
+ })
+_sym_db.RegisterMessage(EdgeLabel)
+
+CountResponse = _reflection.GeneratedProtocolMessageType('CountResponse', (_message.Message,), {
+ 'DESCRIPTOR' : _COUNTRESPONSE,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.CountResponse)
+ })
+_sym_db.RegisterMessage(CountResponse)
+
+StatsRequest = _reflection.GeneratedProtocolMessageType('StatsRequest', (_message.Message,), {
+ 'DESCRIPTOR' : _STATSREQUEST,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.StatsRequest)
+ })
+_sym_db.RegisterMessage(StatsRequest)
+
+StatsResponse = _reflection.GeneratedProtocolMessageType('StatsResponse', (_message.Message,), {
+ 'DESCRIPTOR' : _STATSRESPONSE,
+ '__module__' : 'swh.graph.rpc.swhgraph_pb2'
+ # @@protoc_insertion_point(class_scope:swh.graph.StatsResponse)
+ })
+_sym_db.RegisterMessage(StatsResponse)
+
+_TRAVERSALSERVICE = DESCRIPTOR.services_by_name['TraversalService']
+if _descriptor._USE_C_DESCRIPTORS == False:
+
+ DESCRIPTOR._options = None
+ DESCRIPTOR._serialized_options = b'\n\036org.softwareheritage.graph.rpcB\014GraphServiceP\001'
+ _GRAPHDIRECTION._serialized_start=2853
+ _GRAPHDIRECTION._serialized_end=2896
+ _GETNODEREQUEST._serialized_start=77
+ _GETNODEREQUEST._serialized_end=164
+ _TRAVERSALREQUEST._serialized_start=167
+ _TRAVERSALREQUEST._serialized_end=511
+ _FINDPATHTOREQUEST._serialized_start=514
+ _FINDPATHTOREQUEST._serialized_end=793
+ _FINDPATHBETWEENREQUEST._serialized_start=796
+ _FINDPATHBETWEENREQUEST._serialized_end=1181
+ _NODEFILTER._serialized_start=1184
+ _NODEFILTER._serialized_end=1362
+ _NODE._serialized_start=1365
+ _NODE._serialized_end=1639
+ _PATH._serialized_start=1641
+ _PATH._serialized_end=1726
+ _SUCCESSOR._serialized_start=1728
+ _SUCCESSOR._serialized_end=1806
+ _CONTENTDATA._serialized_start=1808
+ _CONTENTDATA._serialized_end=1893
+ _REVISIONDATA._serialized_start=1896
+ _REVISIONDATA._serialized_end=2222
+ _RELEASEDATA._serialized_start=2225
+ _RELEASEDATA._serialized_end=2430
+ _ORIGINDATA._serialized_start=2432
+ _ORIGINDATA._serialized_end=2470
+ _EDGELABEL._serialized_start=2472
+ _EDGELABEL._serialized_end=2517
+ _COUNTRESPONSE._serialized_start=2519
+ _COUNTRESPONSE._serialized_end=2549
+ _STATSREQUEST._serialized_start=2551
+ _STATSREQUEST._serialized_end=2565
+ _STATSRESPONSE._serialized_start=2568
+ _STATSRESPONSE._serialized_end=2851
+ _TRAVERSALSERVICE._serialized_start=2899
+ _TRAVERSALSERVICE._serialized_end=3362
+# @@protoc_insertion_point(module_scope)
diff --git a/swh/graph/rpc/swhgraph_pb2.pyi b/swh/graph/rpc/swhgraph_pb2.pyi
new file mode 100644
--- /dev/null
+++ b/swh/graph/rpc/swhgraph_pb2.pyi
@@ -0,0 +1,685 @@
+"""
+@generated by mypy-protobuf. Do not edit manually!
+isort:skip_file
+"""
+import builtins
+import google.protobuf.descriptor
+import google.protobuf.field_mask_pb2
+import google.protobuf.internal.containers
+import google.protobuf.internal.enum_type_wrapper
+import google.protobuf.message
+import typing
+import typing_extensions
+
+DESCRIPTOR: google.protobuf.descriptor.FileDescriptor
+
+class _GraphDirection:
+ ValueType = typing.NewType('ValueType', builtins.int)
+ V: typing_extensions.TypeAlias = ValueType
+class _GraphDirectionEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_GraphDirection.ValueType], builtins.type):
+ DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor
+ FORWARD: _GraphDirection.ValueType # 0
+ """Forward DAG: ori -> snp -> rel -> rev -> dir -> cnt"""
+
+ BACKWARD: _GraphDirection.ValueType # 1
+ """Transposed DAG: cnt -> dir -> rev -> rel -> snp -> ori"""
+
+class GraphDirection(_GraphDirection, metaclass=_GraphDirectionEnumTypeWrapper):
+ """Direction of the graph"""
+ pass
+
+FORWARD: GraphDirection.ValueType # 0
+"""Forward DAG: ori -> snp -> rel -> rev -> dir -> cnt"""
+
+BACKWARD: GraphDirection.ValueType # 1
+"""Transposed DAG: cnt -> dir -> rev -> rel -> snp -> ori"""
+
+global___GraphDirection = GraphDirection
+
+
+class GetNodeRequest(google.protobuf.message.Message):
+ """Describe a node to return"""
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ SWHID_FIELD_NUMBER: builtins.int
+ MASK_FIELD_NUMBER: builtins.int
+ swhid: typing.Text
+ """SWHID of the node to return"""
+
+ @property
+ def mask(self) -> google.protobuf.field_mask_pb2.FieldMask:
+ """FieldMask of which fields are to be returned (e.g., "swhid,cnt.length").
+ By default, all fields are returned.
+ """
+ pass
+ def __init__(self,
+ *,
+ swhid: typing.Text = ...,
+ mask: typing.Optional[google.protobuf.field_mask_pb2.FieldMask] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_mask",b"_mask","mask",b"mask"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_mask",b"_mask","mask",b"mask","swhid",b"swhid"]) -> None: ...
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_mask",b"_mask"]) -> typing.Optional[typing_extensions.Literal["mask"]]: ...
+global___GetNodeRequest = GetNodeRequest
+
+class TraversalRequest(google.protobuf.message.Message):
+ """TraversalRequest describes how a breadth-first traversal should be
+ performed, and what should be returned to the client.
+ """
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ SRC_FIELD_NUMBER: builtins.int
+ DIRECTION_FIELD_NUMBER: builtins.int
+ EDGES_FIELD_NUMBER: builtins.int
+ MAX_EDGES_FIELD_NUMBER: builtins.int
+ MIN_DEPTH_FIELD_NUMBER: builtins.int
+ MAX_DEPTH_FIELD_NUMBER: builtins.int
+ RETURN_NODES_FIELD_NUMBER: builtins.int
+ MASK_FIELD_NUMBER: builtins.int
+ @property
+ def src(self) -> google.protobuf.internal.containers.RepeatedScalarFieldContainer[typing.Text]:
+ """Set of source nodes (SWHIDs)"""
+ pass
+ direction: global___GraphDirection.ValueType
+ """Direction of the graph to traverse. Defaults to FORWARD."""
+
+ edges: typing.Text
+ """Edge restriction string (e.g. "rev:dir,dir:cnt").
+ Defaults to "*" (all).
+ """
+
+ max_edges: builtins.int
+ """Maximum number of edges accessed in the traversal, after which it stops.
+ Defaults to infinite.
+ """
+
+ min_depth: builtins.int
+ """Do not return nodes with a depth lower than this number.
+ By default, all depths are returned.
+ """
+
+ max_depth: builtins.int
+ """Maximum depth of the traversal, after which it stops.
+ Defaults to infinite.
+ """
+
+ @property
+ def return_nodes(self) -> global___NodeFilter:
+ """Filter which nodes will be sent to the stream. By default, all nodes are
+ returned.
+ """
+ pass
+ @property
+ def mask(self) -> google.protobuf.field_mask_pb2.FieldMask:
+ """FieldMask of which fields are to be returned (e.g., "swhid,cnt.length").
+ By default, all fields are returned.
+ """
+ pass
+ def __init__(self,
+ *,
+ src: typing.Optional[typing.Iterable[typing.Text]] = ...,
+ direction: global___GraphDirection.ValueType = ...,
+ edges: typing.Optional[typing.Text] = ...,
+ max_edges: typing.Optional[builtins.int] = ...,
+ min_depth: typing.Optional[builtins.int] = ...,
+ max_depth: typing.Optional[builtins.int] = ...,
+ return_nodes: typing.Optional[global___NodeFilter] = ...,
+ mask: typing.Optional[google.protobuf.field_mask_pb2.FieldMask] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_edges",b"_edges","_mask",b"_mask","_max_depth",b"_max_depth","_max_edges",b"_max_edges","_min_depth",b"_min_depth","_return_nodes",b"_return_nodes","edges",b"edges","mask",b"mask","max_depth",b"max_depth","max_edges",b"max_edges","min_depth",b"min_depth","return_nodes",b"return_nodes"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_edges",b"_edges","_mask",b"_mask","_max_depth",b"_max_depth","_max_edges",b"_max_edges","_min_depth",b"_min_depth","_return_nodes",b"_return_nodes","direction",b"direction","edges",b"edges","mask",b"mask","max_depth",b"max_depth","max_edges",b"max_edges","min_depth",b"min_depth","return_nodes",b"return_nodes","src",b"src"]) -> None: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_edges",b"_edges"]) -> typing.Optional[typing_extensions.Literal["edges"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_mask",b"_mask"]) -> typing.Optional[typing_extensions.Literal["mask"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_max_depth",b"_max_depth"]) -> typing.Optional[typing_extensions.Literal["max_depth"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_max_edges",b"_max_edges"]) -> typing.Optional[typing_extensions.Literal["max_edges"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_min_depth",b"_min_depth"]) -> typing.Optional[typing_extensions.Literal["min_depth"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_return_nodes",b"_return_nodes"]) -> typing.Optional[typing_extensions.Literal["return_nodes"]]: ...
+global___TraversalRequest = TraversalRequest
+
+class FindPathToRequest(google.protobuf.message.Message):
+ """FindPathToRequest describes a request to find a shortest path between a
+ set of nodes and a given target criteria, as well as what should be returned
+ in the path.
+ """
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ SRC_FIELD_NUMBER: builtins.int
+ TARGET_FIELD_NUMBER: builtins.int
+ DIRECTION_FIELD_NUMBER: builtins.int
+ EDGES_FIELD_NUMBER: builtins.int
+ MAX_EDGES_FIELD_NUMBER: builtins.int
+ MAX_DEPTH_FIELD_NUMBER: builtins.int
+ MASK_FIELD_NUMBER: builtins.int
+ @property
+ def src(self) -> google.protobuf.internal.containers.RepeatedScalarFieldContainer[typing.Text]:
+ """Set of source nodes (SWHIDs)"""
+ pass
+ @property
+ def target(self) -> global___NodeFilter:
+ """Target criteria, i.e., what constitutes a valid path destination."""
+ pass
+ direction: global___GraphDirection.ValueType
+ """Direction of the graph to traverse. Defaults to FORWARD."""
+
+ edges: typing.Text
+ """Edge restriction string (e.g. "rev:dir,dir:cnt").
+ Defaults to "*" (all).
+ """
+
+ max_edges: builtins.int
+ """Maximum number of edges accessed in the traversal, after which it stops.
+ Defaults to infinite.
+ """
+
+ max_depth: builtins.int
+ """Maximum depth of the traversal, after which it stops.
+ Defaults to infinite.
+ """
+
+ @property
+ def mask(self) -> google.protobuf.field_mask_pb2.FieldMask:
+ """FieldMask of which fields are to be returned (e.g., "swhid,cnt.length").
+ By default, all fields are returned.
+ """
+ pass
+ def __init__(self,
+ *,
+ src: typing.Optional[typing.Iterable[typing.Text]] = ...,
+ target: typing.Optional[global___NodeFilter] = ...,
+ direction: global___GraphDirection.ValueType = ...,
+ edges: typing.Optional[typing.Text] = ...,
+ max_edges: typing.Optional[builtins.int] = ...,
+ max_depth: typing.Optional[builtins.int] = ...,
+ mask: typing.Optional[google.protobuf.field_mask_pb2.FieldMask] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_edges",b"_edges","_mask",b"_mask","_max_depth",b"_max_depth","_max_edges",b"_max_edges","edges",b"edges","mask",b"mask","max_depth",b"max_depth","max_edges",b"max_edges","target",b"target"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_edges",b"_edges","_mask",b"_mask","_max_depth",b"_max_depth","_max_edges",b"_max_edges","direction",b"direction","edges",b"edges","mask",b"mask","max_depth",b"max_depth","max_edges",b"max_edges","src",b"src","target",b"target"]) -> None: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_edges",b"_edges"]) -> typing.Optional[typing_extensions.Literal["edges"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_mask",b"_mask"]) -> typing.Optional[typing_extensions.Literal["mask"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_max_depth",b"_max_depth"]) -> typing.Optional[typing_extensions.Literal["max_depth"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_max_edges",b"_max_edges"]) -> typing.Optional[typing_extensions.Literal["max_edges"]]: ...
+global___FindPathToRequest = FindPathToRequest
+
+class FindPathBetweenRequest(google.protobuf.message.Message):
+ """FindPathToRequest describes a request to find a shortest path between a
+ set of source nodes and a set of destination nodes. It works by performing a
+ bidirectional breadth-first traversal from both sets at the same time.
+ """
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ SRC_FIELD_NUMBER: builtins.int
+ DST_FIELD_NUMBER: builtins.int
+ DIRECTION_FIELD_NUMBER: builtins.int
+ DIRECTION_REVERSE_FIELD_NUMBER: builtins.int
+ EDGES_FIELD_NUMBER: builtins.int
+ EDGES_REVERSE_FIELD_NUMBER: builtins.int
+ MAX_EDGES_FIELD_NUMBER: builtins.int
+ MAX_DEPTH_FIELD_NUMBER: builtins.int
+ MASK_FIELD_NUMBER: builtins.int
+ @property
+ def src(self) -> google.protobuf.internal.containers.RepeatedScalarFieldContainer[typing.Text]:
+ """Set of source nodes (SWHIDs)"""
+ pass
+ @property
+ def dst(self) -> google.protobuf.internal.containers.RepeatedScalarFieldContainer[typing.Text]:
+ """Set of destination nodes (SWHIDs)"""
+ pass
+ direction: global___GraphDirection.ValueType
+ """Direction of the graph to traverse from the source set. Defaults to
+ FORWARD.
+ """
+
+ direction_reverse: global___GraphDirection.ValueType
+ """Direction of the graph to traverse from the destination set. Defaults to
+ the opposite of `direction`. If direction and direction_reverse are
+ identical, it will find the first common successor of both sets in the
+ given direction.
+ """
+
+ edges: typing.Text
+ """Edge restriction string for the traversal from the source set.
+ (e.g. "rev:dir,dir:cnt"). Defaults to "*" (all).
+ """
+
+ edges_reverse: typing.Text
+ """Edge restriction string for the reverse traversal from the destination
+ set.
+ If not specified:
+ - If `edges` is not specified either, defaults to "*"
+ - If direction == direction_reverse, defaults to `edges`
+ - If direction != direction_reverse, defaults to the reverse of `edges`
+ (e.g. "rev:dir" becomes "dir:rev").
+ """
+
+ max_edges: builtins.int
+ """Maximum number of edges accessed in the traversal, after which it stops.
+ Defaults to infinite.
+ """
+
+ max_depth: builtins.int
+ """Maximum depth of the traversal, after which it stops.
+ Defaults to infinite.
+ """
+
+ @property
+ def mask(self) -> google.protobuf.field_mask_pb2.FieldMask:
+ """FieldMask of which fields are to be returned (e.g., "swhid,cnt.length").
+ By default, all fields are returned.
+ """
+ pass
+ def __init__(self,
+ *,
+ src: typing.Optional[typing.Iterable[typing.Text]] = ...,
+ dst: typing.Optional[typing.Iterable[typing.Text]] = ...,
+ direction: global___GraphDirection.ValueType = ...,
+ direction_reverse: typing.Optional[global___GraphDirection.ValueType] = ...,
+ edges: typing.Optional[typing.Text] = ...,
+ edges_reverse: typing.Optional[typing.Text] = ...,
+ max_edges: typing.Optional[builtins.int] = ...,
+ max_depth: typing.Optional[builtins.int] = ...,
+ mask: typing.Optional[google.protobuf.field_mask_pb2.FieldMask] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_direction_reverse",b"_direction_reverse","_edges",b"_edges","_edges_reverse",b"_edges_reverse","_mask",b"_mask","_max_depth",b"_max_depth","_max_edges",b"_max_edges","direction_reverse",b"direction_reverse","edges",b"edges","edges_reverse",b"edges_reverse","mask",b"mask","max_depth",b"max_depth","max_edges",b"max_edges"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_direction_reverse",b"_direction_reverse","_edges",b"_edges","_edges_reverse",b"_edges_reverse","_mask",b"_mask","_max_depth",b"_max_depth","_max_edges",b"_max_edges","direction",b"direction","direction_reverse",b"direction_reverse","dst",b"dst","edges",b"edges","edges_reverse",b"edges_reverse","mask",b"mask","max_depth",b"max_depth","max_edges",b"max_edges","src",b"src"]) -> None: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_direction_reverse",b"_direction_reverse"]) -> typing.Optional[typing_extensions.Literal["direction_reverse"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_edges",b"_edges"]) -> typing.Optional[typing_extensions.Literal["edges"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_edges_reverse",b"_edges_reverse"]) -> typing.Optional[typing_extensions.Literal["edges_reverse"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_mask",b"_mask"]) -> typing.Optional[typing_extensions.Literal["mask"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_max_depth",b"_max_depth"]) -> typing.Optional[typing_extensions.Literal["max_depth"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_max_edges",b"_max_edges"]) -> typing.Optional[typing_extensions.Literal["max_edges"]]: ...
+global___FindPathBetweenRequest = FindPathBetweenRequest
+
+class NodeFilter(google.protobuf.message.Message):
+ """Represents various criteria that make a given node "valid". A node is
+ only valid if all the subcriteria present in this message are fulfilled.
+ """
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ TYPES_FIELD_NUMBER: builtins.int
+ MIN_TRAVERSAL_SUCCESSORS_FIELD_NUMBER: builtins.int
+ MAX_TRAVERSAL_SUCCESSORS_FIELD_NUMBER: builtins.int
+ types: typing.Text
+ """Node restriction string. (e.g. "dir,cnt,rev"). Defaults to "*" (all)."""
+
+ min_traversal_successors: builtins.int
+ """Minimum number of successors encountered *during the traversal*.
+ Default: no constraint
+ """
+
+ max_traversal_successors: builtins.int
+ """Maximum number of successors encountered *during the traversal*.
+ Default: no constraint
+ """
+
+ def __init__(self,
+ *,
+ types: typing.Optional[typing.Text] = ...,
+ min_traversal_successors: typing.Optional[builtins.int] = ...,
+ max_traversal_successors: typing.Optional[builtins.int] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_max_traversal_successors",b"_max_traversal_successors","_min_traversal_successors",b"_min_traversal_successors","_types",b"_types","max_traversal_successors",b"max_traversal_successors","min_traversal_successors",b"min_traversal_successors","types",b"types"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_max_traversal_successors",b"_max_traversal_successors","_min_traversal_successors",b"_min_traversal_successors","_types",b"_types","max_traversal_successors",b"max_traversal_successors","min_traversal_successors",b"min_traversal_successors","types",b"types"]) -> None: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_max_traversal_successors",b"_max_traversal_successors"]) -> typing.Optional[typing_extensions.Literal["max_traversal_successors"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_min_traversal_successors",b"_min_traversal_successors"]) -> typing.Optional[typing_extensions.Literal["min_traversal_successors"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_types",b"_types"]) -> typing.Optional[typing_extensions.Literal["types"]]: ...
+global___NodeFilter = NodeFilter
+
+class Node(google.protobuf.message.Message):
+ """Represents a node in the graph."""
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ SWHID_FIELD_NUMBER: builtins.int
+ SUCCESSOR_FIELD_NUMBER: builtins.int
+ NUM_SUCCESSORS_FIELD_NUMBER: builtins.int
+ CNT_FIELD_NUMBER: builtins.int
+ REV_FIELD_NUMBER: builtins.int
+ REL_FIELD_NUMBER: builtins.int
+ ORI_FIELD_NUMBER: builtins.int
+ swhid: typing.Text
+ """The SWHID of the graph node."""
+
+ @property
+ def successor(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___Successor]:
+ """List of relevant successors of this node."""
+ pass
+ num_successors: builtins.int
+ """Number of relevant successors."""
+
+ @property
+ def cnt(self) -> global___ContentData: ...
+ @property
+ def rev(self) -> global___RevisionData: ...
+ @property
+ def rel(self) -> global___ReleaseData: ...
+ @property
+ def ori(self) -> global___OriginData: ...
+ def __init__(self,
+ *,
+ swhid: typing.Text = ...,
+ successor: typing.Optional[typing.Iterable[global___Successor]] = ...,
+ num_successors: typing.Optional[builtins.int] = ...,
+ cnt: typing.Optional[global___ContentData] = ...,
+ rev: typing.Optional[global___RevisionData] = ...,
+ rel: typing.Optional[global___ReleaseData] = ...,
+ ori: typing.Optional[global___OriginData] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_num_successors",b"_num_successors","cnt",b"cnt","data",b"data","num_successors",b"num_successors","ori",b"ori","rel",b"rel","rev",b"rev"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_num_successors",b"_num_successors","cnt",b"cnt","data",b"data","num_successors",b"num_successors","ori",b"ori","rel",b"rel","rev",b"rev","successor",b"successor","swhid",b"swhid"]) -> None: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_num_successors",b"_num_successors"]) -> typing.Optional[typing_extensions.Literal["num_successors"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["data",b"data"]) -> typing.Optional[typing_extensions.Literal["cnt","rev","rel","ori"]]: ...
+global___Node = Node
+
+class Path(google.protobuf.message.Message):
+ """Represents a path in the graph."""
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ NODE_FIELD_NUMBER: builtins.int
+ MIDPOINT_INDEX_FIELD_NUMBER: builtins.int
+ @property
+ def node(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___Node]:
+ """List of nodes in the path, from source to destination"""
+ pass
+ midpoint_index: builtins.int
+ """Index of the "midpoint" of the path. For paths obtained with
+ bidirectional search queries, this is the node that joined the two
+ sets together. When looking for a common ancestor between two nodes by
+ performing a FindPathBetween search with two backward graphs, this will
+ be the index of the common ancestor in the path.
+ """
+
+ def __init__(self,
+ *,
+ node: typing.Optional[typing.Iterable[global___Node]] = ...,
+ midpoint_index: typing.Optional[builtins.int] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_midpoint_index",b"_midpoint_index","midpoint_index",b"midpoint_index"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_midpoint_index",b"_midpoint_index","midpoint_index",b"midpoint_index","node",b"node"]) -> None: ...
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_midpoint_index",b"_midpoint_index"]) -> typing.Optional[typing_extensions.Literal["midpoint_index"]]: ...
+global___Path = Path
+
+class Successor(google.protobuf.message.Message):
+ """Represents a successor of a given node."""
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ SWHID_FIELD_NUMBER: builtins.int
+ LABEL_FIELD_NUMBER: builtins.int
+ swhid: typing.Text
+ """The SWHID of the successor"""
+
+ @property
+ def label(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___EdgeLabel]:
+ """A list of edge labels for the given edge"""
+ pass
+ def __init__(self,
+ *,
+ swhid: typing.Optional[typing.Text] = ...,
+ label: typing.Optional[typing.Iterable[global___EdgeLabel]] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_swhid",b"_swhid","swhid",b"swhid"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_swhid",b"_swhid","label",b"label","swhid",b"swhid"]) -> None: ...
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_swhid",b"_swhid"]) -> typing.Optional[typing_extensions.Literal["swhid"]]: ...
+global___Successor = Successor
+
+class ContentData(google.protobuf.message.Message):
+ """Content node properties"""
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ LENGTH_FIELD_NUMBER: builtins.int
+ IS_SKIPPED_FIELD_NUMBER: builtins.int
+ length: builtins.int
+ """Length of the blob, in bytes"""
+
+ is_skipped: builtins.bool
+ """Whether the content was skipped during ingestion."""
+
+ def __init__(self,
+ *,
+ length: typing.Optional[builtins.int] = ...,
+ is_skipped: typing.Optional[builtins.bool] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_is_skipped",b"_is_skipped","_length",b"_length","is_skipped",b"is_skipped","length",b"length"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_is_skipped",b"_is_skipped","_length",b"_length","is_skipped",b"is_skipped","length",b"length"]) -> None: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_is_skipped",b"_is_skipped"]) -> typing.Optional[typing_extensions.Literal["is_skipped"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_length",b"_length"]) -> typing.Optional[typing_extensions.Literal["length"]]: ...
+global___ContentData = ContentData
+
+class RevisionData(google.protobuf.message.Message):
+ """Revision node properties"""
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ AUTHOR_FIELD_NUMBER: builtins.int
+ AUTHOR_DATE_FIELD_NUMBER: builtins.int
+ AUTHOR_DATE_OFFSET_FIELD_NUMBER: builtins.int
+ COMMITTER_FIELD_NUMBER: builtins.int
+ COMMITTER_DATE_FIELD_NUMBER: builtins.int
+ COMMITTER_DATE_OFFSET_FIELD_NUMBER: builtins.int
+ MESSAGE_FIELD_NUMBER: builtins.int
+ author: builtins.int
+ """Revision author ID (anonymized)"""
+
+ author_date: builtins.int
+ """UNIX timestamp of the revision date (UTC)"""
+
+ author_date_offset: builtins.int
+ """Timezone of the revision author date as an offset from UTC"""
+
+ committer: builtins.int
+ """Revision committer ID (anonymized)"""
+
+ committer_date: builtins.int
+ """UNIX timestamp of the revision committer date (UTC)"""
+
+ committer_date_offset: builtins.int
+ """Timezone of the revision committer date as an offset from UTC"""
+
+ message: builtins.bytes
+ """Revision message"""
+
+ def __init__(self,
+ *,
+ author: typing.Optional[builtins.int] = ...,
+ author_date: typing.Optional[builtins.int] = ...,
+ author_date_offset: typing.Optional[builtins.int] = ...,
+ committer: typing.Optional[builtins.int] = ...,
+ committer_date: typing.Optional[builtins.int] = ...,
+ committer_date_offset: typing.Optional[builtins.int] = ...,
+ message: typing.Optional[builtins.bytes] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_author",b"_author","_author_date",b"_author_date","_author_date_offset",b"_author_date_offset","_committer",b"_committer","_committer_date",b"_committer_date","_committer_date_offset",b"_committer_date_offset","_message",b"_message","author",b"author","author_date",b"author_date","author_date_offset",b"author_date_offset","committer",b"committer","committer_date",b"committer_date","committer_date_offset",b"committer_date_offset","message",b"message"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_author",b"_author","_author_date",b"_author_date","_author_date_offset",b"_author_date_offset","_committer",b"_committer","_committer_date",b"_committer_date","_committer_date_offset",b"_committer_date_offset","_message",b"_message","author",b"author","author_date",b"author_date","author_date_offset",b"author_date_offset","committer",b"committer","committer_date",b"committer_date","committer_date_offset",b"committer_date_offset","message",b"message"]) -> None: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_author",b"_author"]) -> typing.Optional[typing_extensions.Literal["author"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_author_date",b"_author_date"]) -> typing.Optional[typing_extensions.Literal["author_date"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_author_date_offset",b"_author_date_offset"]) -> typing.Optional[typing_extensions.Literal["author_date_offset"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_committer",b"_committer"]) -> typing.Optional[typing_extensions.Literal["committer"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_committer_date",b"_committer_date"]) -> typing.Optional[typing_extensions.Literal["committer_date"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_committer_date_offset",b"_committer_date_offset"]) -> typing.Optional[typing_extensions.Literal["committer_date_offset"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_message",b"_message"]) -> typing.Optional[typing_extensions.Literal["message"]]: ...
+global___RevisionData = RevisionData
+
+class ReleaseData(google.protobuf.message.Message):
+ """Release node properties"""
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ AUTHOR_FIELD_NUMBER: builtins.int
+ AUTHOR_DATE_FIELD_NUMBER: builtins.int
+ AUTHOR_DATE_OFFSET_FIELD_NUMBER: builtins.int
+ NAME_FIELD_NUMBER: builtins.int
+ MESSAGE_FIELD_NUMBER: builtins.int
+ author: builtins.int
+ """Release author ID (anonymized)"""
+
+ author_date: builtins.int
+ """UNIX timestamp of the release date (UTC)"""
+
+ author_date_offset: builtins.int
+ """Timezone of the release author date as an offset from UTC"""
+
+ name: builtins.bytes
+ """Release name"""
+
+ message: builtins.bytes
+ """Release message"""
+
+ def __init__(self,
+ *,
+ author: typing.Optional[builtins.int] = ...,
+ author_date: typing.Optional[builtins.int] = ...,
+ author_date_offset: typing.Optional[builtins.int] = ...,
+ name: typing.Optional[builtins.bytes] = ...,
+ message: typing.Optional[builtins.bytes] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_author",b"_author","_author_date",b"_author_date","_author_date_offset",b"_author_date_offset","_message",b"_message","_name",b"_name","author",b"author","author_date",b"author_date","author_date_offset",b"author_date_offset","message",b"message","name",b"name"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_author",b"_author","_author_date",b"_author_date","_author_date_offset",b"_author_date_offset","_message",b"_message","_name",b"_name","author",b"author","author_date",b"author_date","author_date_offset",b"author_date_offset","message",b"message","name",b"name"]) -> None: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_author",b"_author"]) -> typing.Optional[typing_extensions.Literal["author"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_author_date",b"_author_date"]) -> typing.Optional[typing_extensions.Literal["author_date"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_author_date_offset",b"_author_date_offset"]) -> typing.Optional[typing_extensions.Literal["author_date_offset"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_message",b"_message"]) -> typing.Optional[typing_extensions.Literal["message"]]: ...
+ @typing.overload
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_name",b"_name"]) -> typing.Optional[typing_extensions.Literal["name"]]: ...
+global___ReleaseData = ReleaseData
+
+class OriginData(google.protobuf.message.Message):
+ """Origin node properties"""
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ URL_FIELD_NUMBER: builtins.int
+ url: typing.Text
+ """URL of the origin"""
+
+ def __init__(self,
+ *,
+ url: typing.Optional[typing.Text] = ...,
+ ) -> None: ...
+ def HasField(self, field_name: typing_extensions.Literal["_url",b"_url","url",b"url"]) -> builtins.bool: ...
+ def ClearField(self, field_name: typing_extensions.Literal["_url",b"_url","url",b"url"]) -> None: ...
+ def WhichOneof(self, oneof_group: typing_extensions.Literal["_url",b"_url"]) -> typing.Optional[typing_extensions.Literal["url"]]: ...
+global___OriginData = OriginData
+
+class EdgeLabel(google.protobuf.message.Message):
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ NAME_FIELD_NUMBER: builtins.int
+ PERMISSION_FIELD_NUMBER: builtins.int
+ name: builtins.bytes
+ """Directory entry name for directories, branch name for snapshots"""
+
+ permission: builtins.int
+ """Entry permission (only set for directories)."""
+
+ def __init__(self,
+ *,
+ name: builtins.bytes = ...,
+ permission: builtins.int = ...,
+ ) -> None: ...
+ def ClearField(self, field_name: typing_extensions.Literal["name",b"name","permission",b"permission"]) -> None: ...
+global___EdgeLabel = EdgeLabel
+
+class CountResponse(google.protobuf.message.Message):
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ COUNT_FIELD_NUMBER: builtins.int
+ count: builtins.int
+ def __init__(self,
+ *,
+ count: builtins.int = ...,
+ ) -> None: ...
+ def ClearField(self, field_name: typing_extensions.Literal["count",b"count"]) -> None: ...
+global___CountResponse = CountResponse
+
+class StatsRequest(google.protobuf.message.Message):
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ def __init__(self,
+ ) -> None: ...
+global___StatsRequest = StatsRequest
+
+class StatsResponse(google.protobuf.message.Message):
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+ NUM_NODES_FIELD_NUMBER: builtins.int
+ NUM_EDGES_FIELD_NUMBER: builtins.int
+ COMPRESSION_RATIO_FIELD_NUMBER: builtins.int
+ BITS_PER_NODE_FIELD_NUMBER: builtins.int
+ BITS_PER_EDGE_FIELD_NUMBER: builtins.int
+ AVG_LOCALITY_FIELD_NUMBER: builtins.int
+ INDEGREE_MIN_FIELD_NUMBER: builtins.int
+ INDEGREE_MAX_FIELD_NUMBER: builtins.int
+ INDEGREE_AVG_FIELD_NUMBER: builtins.int
+ OUTDEGREE_MIN_FIELD_NUMBER: builtins.int
+ OUTDEGREE_MAX_FIELD_NUMBER: builtins.int
+ OUTDEGREE_AVG_FIELD_NUMBER: builtins.int
+ num_nodes: builtins.int
+ """Number of nodes in the graph"""
+
+ num_edges: builtins.int
+ """Number of edges in the graph"""
+
+ compression_ratio: builtins.float
+ """Ratio between the graph size and the information-theoretical lower
+ bound
+ """
+
+ bits_per_node: builtins.float
+ """Number of bits per node (overall graph size in bits divided by the
+ number of nodes)
+ """
+
+ bits_per_edge: builtins.float
+ """Number of bits per edge (overall graph size in bits divided by the
+ number of arcs).
+ """
+
+ avg_locality: builtins.float
+ indegree_min: builtins.int
+ """Smallest indegree"""
+
+ indegree_max: builtins.int
+ """Largest indegree"""
+
+ indegree_avg: builtins.float
+ """Average indegree"""
+
+ outdegree_min: builtins.int
+ """Smallest outdegree"""
+
+ outdegree_max: builtins.int
+ """Largest outdegree"""
+
+ outdegree_avg: builtins.float
+ """Average outdegree"""
+
+ def __init__(self,
+ *,
+ num_nodes: builtins.int = ...,
+ num_edges: builtins.int = ...,
+ compression_ratio: builtins.float = ...,
+ bits_per_node: builtins.float = ...,
+ bits_per_edge: builtins.float = ...,
+ avg_locality: builtins.float = ...,
+ indegree_min: builtins.int = ...,
+ indegree_max: builtins.int = ...,
+ indegree_avg: builtins.float = ...,
+ outdegree_min: builtins.int = ...,
+ outdegree_max: builtins.int = ...,
+ outdegree_avg: builtins.float = ...,
+ ) -> None: ...
+ def ClearField(self, field_name: typing_extensions.Literal["avg_locality",b"avg_locality","bits_per_edge",b"bits_per_edge","bits_per_node",b"bits_per_node","compression_ratio",b"compression_ratio","indegree_avg",b"indegree_avg","indegree_max",b"indegree_max","indegree_min",b"indegree_min","num_edges",b"num_edges","num_nodes",b"num_nodes","outdegree_avg",b"outdegree_avg","outdegree_max",b"outdegree_max","outdegree_min",b"outdegree_min"]) -> None: ...
+global___StatsResponse = StatsResponse
diff --git a/swh/graph/rpc/swhgraph_pb2_grpc.py b/swh/graph/rpc/swhgraph_pb2_grpc.py
new file mode 100644
--- /dev/null
+++ b/swh/graph/rpc/swhgraph_pb2_grpc.py
@@ -0,0 +1,303 @@
+# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
+"""Client and server classes corresponding to protobuf-defined services."""
+import grpc
+
+from swh.graph.rpc import swhgraph_pb2 as swh_dot_graph_dot_rpc_dot_swhgraph__pb2
+
+
+class TraversalServiceStub(object):
+ """Graph traversal service
+ """
+
+ def __init__(self, channel):
+ """Constructor.
+
+ Args:
+ channel: A grpc.Channel.
+ """
+ self.GetNode = channel.unary_unary(
+ '/swh.graph.TraversalService/GetNode',
+ request_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.GetNodeRequest.SerializeToString,
+ response_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Node.FromString,
+ )
+ self.Traverse = channel.unary_stream(
+ '/swh.graph.TraversalService/Traverse',
+ request_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.TraversalRequest.SerializeToString,
+ response_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Node.FromString,
+ )
+ self.FindPathTo = channel.unary_unary(
+ '/swh.graph.TraversalService/FindPathTo',
+ request_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.FindPathToRequest.SerializeToString,
+ response_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Path.FromString,
+ )
+ self.FindPathBetween = channel.unary_unary(
+ '/swh.graph.TraversalService/FindPathBetween',
+ request_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.FindPathBetweenRequest.SerializeToString,
+ response_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Path.FromString,
+ )
+ self.CountNodes = channel.unary_unary(
+ '/swh.graph.TraversalService/CountNodes',
+ request_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.TraversalRequest.SerializeToString,
+ response_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.CountResponse.FromString,
+ )
+ self.CountEdges = channel.unary_unary(
+ '/swh.graph.TraversalService/CountEdges',
+ request_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.TraversalRequest.SerializeToString,
+ response_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.CountResponse.FromString,
+ )
+ self.Stats = channel.unary_unary(
+ '/swh.graph.TraversalService/Stats',
+ request_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.StatsRequest.SerializeToString,
+ response_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.StatsResponse.FromString,
+ )
+
+
+class TraversalServiceServicer(object):
+ """Graph traversal service
+ """
+
+ def GetNode(self, request, context):
+ """GetNode returns a single Node and its properties.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details('Method not implemented!')
+ raise NotImplementedError('Method not implemented!')
+
+ def Traverse(self, request, context):
+ """Traverse performs a breadth-first graph traversal from a set of source
+ nodes, then streams the nodes it encounters (if they match a given
+ return filter), along with their properties.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details('Method not implemented!')
+ raise NotImplementedError('Method not implemented!')
+
+ def FindPathTo(self, request, context):
+ """FindPathTo searches for a shortest path between a set of source nodes
+ and a node that matches a specific *criteria*.
+
+ It does so by performing a breadth-first search from the source node,
+ until any node that matches the given criteria is found, then follows
+ back its parents to return a shortest path from the source set to that
+ node.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details('Method not implemented!')
+ raise NotImplementedError('Method not implemented!')
+
+ def FindPathBetween(self, request, context):
+ """FindPathBetween searches for a shortest path between a set of source
+ nodes and a set of destination nodes.
+
+ It does so by performing a *bidirectional breadth-first search*, i.e.,
+ two parallel breadth-first searches, one from the source set ("src-BFS")
+ and one from the destination set ("dst-BFS"), until both searches find a
+ common node that joins their visited sets. This node is called the
+ "midpoint node".
+ The path returned is the path src -> ... -> midpoint -> ... -> dst,
+ which is always a shortest path between src and dst.
+
+ The graph direction of both BFS can be configured separately. By
+ default, the dst-BFS will use the graph in the opposite direction than
+ the src-BFS (if direction = FORWARD, by default direction_reverse =
+ BACKWARD, and vice-versa). The default behavior is thus to search for
+ a shortest path between two nodes in a given direction. However, one
+ can also specify FORWARD or BACKWARD for *both* the src-BFS and the
+ dst-BFS. This will search for a common descendant or a common ancestor
+ between the two sets, respectively. These will be the midpoints of the
+ returned path.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details('Method not implemented!')
+ raise NotImplementedError('Method not implemented!')
+
+ def CountNodes(self, request, context):
+ """CountNodes does the same as Traverse, but only returns the number of
+ nodes accessed during the traversal.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details('Method not implemented!')
+ raise NotImplementedError('Method not implemented!')
+
+ def CountEdges(self, request, context):
+ """CountEdges does the same as Traverse, but only returns the number of
+ edges accessed during the traversal.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details('Method not implemented!')
+ raise NotImplementedError('Method not implemented!')
+
+ def Stats(self, request, context):
+ """Stats returns various statistics on the overall graph.
+ """
+ context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+ context.set_details('Method not implemented!')
+ raise NotImplementedError('Method not implemented!')
+
+
+def add_TraversalServiceServicer_to_server(servicer, server):
+ rpc_method_handlers = {
+ 'GetNode': grpc.unary_unary_rpc_method_handler(
+ servicer.GetNode,
+ request_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.GetNodeRequest.FromString,
+ response_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Node.SerializeToString,
+ ),
+ 'Traverse': grpc.unary_stream_rpc_method_handler(
+ servicer.Traverse,
+ request_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.TraversalRequest.FromString,
+ response_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Node.SerializeToString,
+ ),
+ 'FindPathTo': grpc.unary_unary_rpc_method_handler(
+ servicer.FindPathTo,
+ request_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.FindPathToRequest.FromString,
+ response_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Path.SerializeToString,
+ ),
+ 'FindPathBetween': grpc.unary_unary_rpc_method_handler(
+ servicer.FindPathBetween,
+ request_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.FindPathBetweenRequest.FromString,
+ response_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Path.SerializeToString,
+ ),
+ 'CountNodes': grpc.unary_unary_rpc_method_handler(
+ servicer.CountNodes,
+ request_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.TraversalRequest.FromString,
+ response_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.CountResponse.SerializeToString,
+ ),
+ 'CountEdges': grpc.unary_unary_rpc_method_handler(
+ servicer.CountEdges,
+ request_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.TraversalRequest.FromString,
+ response_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.CountResponse.SerializeToString,
+ ),
+ 'Stats': grpc.unary_unary_rpc_method_handler(
+ servicer.Stats,
+ request_deserializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.StatsRequest.FromString,
+ response_serializer=swh_dot_graph_dot_rpc_dot_swhgraph__pb2.StatsResponse.SerializeToString,
+ ),
+ }
+ generic_handler = grpc.method_handlers_generic_handler(
+ 'swh.graph.TraversalService', rpc_method_handlers)
+ server.add_generic_rpc_handlers((generic_handler,))
+
+
+ # This class is part of an EXPERIMENTAL API.
+class TraversalService(object):
+ """Graph traversal service
+ """
+
+ @staticmethod
+ def GetNode(request,
+ target,
+ options=(),
+ channel_credentials=None,
+ call_credentials=None,
+ insecure=False,
+ compression=None,
+ wait_for_ready=None,
+ timeout=None,
+ metadata=None):
+ return grpc.experimental.unary_unary(request, target, '/swh.graph.TraversalService/GetNode',
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.GetNodeRequest.SerializeToString,
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Node.FromString,
+ options, channel_credentials,
+ insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+ @staticmethod
+ def Traverse(request,
+ target,
+ options=(),
+ channel_credentials=None,
+ call_credentials=None,
+ insecure=False,
+ compression=None,
+ wait_for_ready=None,
+ timeout=None,
+ metadata=None):
+ return grpc.experimental.unary_stream(request, target, '/swh.graph.TraversalService/Traverse',
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.TraversalRequest.SerializeToString,
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Node.FromString,
+ options, channel_credentials,
+ insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+ @staticmethod
+ def FindPathTo(request,
+ target,
+ options=(),
+ channel_credentials=None,
+ call_credentials=None,
+ insecure=False,
+ compression=None,
+ wait_for_ready=None,
+ timeout=None,
+ metadata=None):
+ return grpc.experimental.unary_unary(request, target, '/swh.graph.TraversalService/FindPathTo',
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.FindPathToRequest.SerializeToString,
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Path.FromString,
+ options, channel_credentials,
+ insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+ @staticmethod
+ def FindPathBetween(request,
+ target,
+ options=(),
+ channel_credentials=None,
+ call_credentials=None,
+ insecure=False,
+ compression=None,
+ wait_for_ready=None,
+ timeout=None,
+ metadata=None):
+ return grpc.experimental.unary_unary(request, target, '/swh.graph.TraversalService/FindPathBetween',
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.FindPathBetweenRequest.SerializeToString,
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.Path.FromString,
+ options, channel_credentials,
+ insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+ @staticmethod
+ def CountNodes(request,
+ target,
+ options=(),
+ channel_credentials=None,
+ call_credentials=None,
+ insecure=False,
+ compression=None,
+ wait_for_ready=None,
+ timeout=None,
+ metadata=None):
+ return grpc.experimental.unary_unary(request, target, '/swh.graph.TraversalService/CountNodes',
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.TraversalRequest.SerializeToString,
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.CountResponse.FromString,
+ options, channel_credentials,
+ insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+ @staticmethod
+ def CountEdges(request,
+ target,
+ options=(),
+ channel_credentials=None,
+ call_credentials=None,
+ insecure=False,
+ compression=None,
+ wait_for_ready=None,
+ timeout=None,
+ metadata=None):
+ return grpc.experimental.unary_unary(request, target, '/swh.graph.TraversalService/CountEdges',
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.TraversalRequest.SerializeToString,
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.CountResponse.FromString,
+ options, channel_credentials,
+ insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
+
+ @staticmethod
+ def Stats(request,
+ target,
+ options=(),
+ channel_credentials=None,
+ call_credentials=None,
+ insecure=False,
+ compression=None,
+ wait_for_ready=None,
+ timeout=None,
+ metadata=None):
+ return grpc.experimental.unary_unary(request, target, '/swh.graph.TraversalService/Stats',
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.StatsRequest.SerializeToString,
+ swh_dot_graph_dot_rpc_dot_swhgraph__pb2.StatsResponse.FromString,
+ options, channel_credentials,
+ insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
diff --git a/swh/graph/rpc_server.py b/swh/graph/rpc_server.py
new file mode 100644
--- /dev/null
+++ b/swh/graph/rpc_server.py
@@ -0,0 +1,33 @@
+# Copyright (C) 2021 The Software Heritage developers
+# See the AUTHORS file at the top-level directory of this distribution
+# License: GNU General Public License version 3, or any later version
+# See top-level LICENSE file for more information
+
+"""
+A simple tool to start the swh-graph GRPC server in Java.
+"""
+
+import subprocess
+
+import aiohttp.test_utils
+import aiohttp.web
+
+from swh.graph.config import check_config
+
+
+def spawn_java_rpc_server(config, port=None):
+ if port is None:
+ port = aiohttp.test_utils.unused_port()
+ config = check_config(config or {})
+ cmd = [
+ "java",
+ "-cp",
+ config["classpath"],
+ *config["java_tool_options"].split(),
+ "org.softwareheritage.graph.rpc.GraphServer",
+ "--port",
+ str(port),
+ config["graph"]["path"],
+ ]
+ server = subprocess.Popen(cmd)
+ return server, port
diff --git a/swh/graph/server/__init__.py b/swh/graph/server/__init__.py
deleted file mode 100644
diff --git a/swh/graph/tests/conftest.py b/swh/graph/tests/conftest.py
--- a/swh/graph/tests/conftest.py
+++ b/swh/graph/tests/conftest.py
@@ -10,8 +10,8 @@
from aiohttp.test_utils import TestClient, TestServer, loop_context
import pytest
-from swh.graph.client import RemoteGraphClient
-from swh.graph.naive_client import NaiveClient
+from swh.graph.http_client import RemoteGraphClient
+from swh.graph.http_naive_client import NaiveClient
SWH_GRAPH_TESTS_ROOT = Path(__file__).parents[0]
TEST_GRAPH_PATH = SWH_GRAPH_TESTS_ROOT / "dataset/compressed/example"
@@ -24,13 +24,12 @@
def run(self):
# Lazy import to allow debian packaging
- from swh.graph.backend import Backend
- from swh.graph.server.app import make_app
+ from swh.graph.http_server import make_app
try:
- backend = Backend(graph_path=str(TEST_GRAPH_PATH))
+ config = {"graph": {"path": TEST_GRAPH_PATH}}
with loop_context() as loop:
- app = make_app(backend=backend, debug=True)
+ app = make_app(config=config, debug=True)
client = TestClient(TestServer(app), loop=loop)
loop.run_until_complete(client.start_server())
url = client.make_url("/graph/")
diff --git a/swh/graph/tests/test_api_client.py b/swh/graph/tests/test_http_client.py
rename from swh/graph/tests/test_api_client.py
rename to swh/graph/tests/test_http_client.py
--- a/swh/graph/tests/test_api_client.py
+++ b/swh/graph/tests/test_http_client.py
@@ -4,7 +4,7 @@
from pytest import raises
from swh.core.api import RemoteException
-from swh.graph.client import GraphArgumentException
+from swh.graph.http_client import GraphArgumentException
TEST_ORIGIN_ID = "swh:1:ori:{}".format(
hashlib.sha1(b"https://example.com/swh/graph").hexdigest()
@@ -13,31 +13,18 @@
def test_stats(graph_client):
stats = graph_client.stats()
-
- assert set(stats.keys()) == {"counts", "ratios", "indegree", "outdegree"}
-
- assert set(stats["counts"].keys()) == {"nodes", "edges"}
- assert set(stats["ratios"].keys()) == {
- "compression",
- "bits_per_node",
- "bits_per_edge",
- "avg_locality",
- }
- assert set(stats["indegree"].keys()) == {"min", "max", "avg"}
- assert set(stats["outdegree"].keys()) == {"min", "max", "avg"}
-
- assert stats["counts"]["nodes"] == 21
- assert stats["counts"]["edges"] == 23
- assert isinstance(stats["ratios"]["compression"], float)
- assert isinstance(stats["ratios"]["bits_per_node"], float)
- assert isinstance(stats["ratios"]["bits_per_edge"], float)
- assert isinstance(stats["ratios"]["avg_locality"], float)
- assert stats["indegree"]["min"] == 0
- assert stats["indegree"]["max"] == 3
- assert isinstance(stats["indegree"]["avg"], float)
- assert stats["outdegree"]["min"] == 0
- assert stats["outdegree"]["max"] == 3
- assert isinstance(stats["outdegree"]["avg"], float)
+ assert stats["num_nodes"] == 21
+ assert stats["num_edges"] == 23
+ assert isinstance(stats["compression_ratio"], float)
+ assert isinstance(stats["bits_per_node"], float)
+ assert isinstance(stats["bits_per_edge"], float)
+ assert isinstance(stats["avg_locality"], float)
+ assert stats["indegree_min"] == 0
+ assert stats["indegree_max"] == 3
+ assert isinstance(stats["indegree_avg"], float)
+ assert stats["outdegree_min"] == 0
+ assert stats["outdegree_max"] == 3
+ assert isinstance(stats["outdegree_avg"], float)
def test_leaves(graph_client):
@@ -259,6 +246,7 @@
assert set(actual) == set(expected)
+@pytest.mark.skip(reason="Random walk is deprecated")
def test_random_walk_dst_is_type(graph_client):
"""as the walk is random, we test a visit from a cnt node to a release
reachable from every single path in the backward graph, and only check the
@@ -288,6 +276,7 @@
assert len(actual) == 3
+@pytest.mark.skip(reason="Random walk is deprecated")
def test_random_walk_dst_is_node(graph_client):
"""Same as test_random_walk_dst_is_type, but we target the specific release
node instead of a type
File Metadata
Details
Attached
Mime Type
text/plain
Expires
Nov 5 2024, 6:47 AM (8 w, 2 d ago)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
3222901
Attached To
D7890: Migrate low-level RPC API from Py4J to GRPC
Event Timeline
Log In to Comment