I'm approving this as it's good enough. But please make the seed a real random seed rather than hard coded (in a subsequent commit). It's *useful* to have a fixed seed for reproducibility, but it should not be the default and it should be possible to pass it externally, e.g., as a class parameter and/or CLI option.

• we should wrap the JSON return type more properly. Please use "result" instead of "content", everything else (like "timings" now, possibly other stuff in the future) will be metadata about the result

True, I will also rename "timings" into "metadata" because I don't like having both "Timing" and "Timings" class name.

I hadn't in mind to lift the timings up to the REST layer, but why not, it will enable doing interesting stuff using non Java clients. However, a couple of change requests:

In T1234#36008, @ardumont wrote:

In the end the PGPASSFILE stuff cannot work with one file globally defined.
Because for some reason, the authors decided that it needs to be only user readable...

Just wanted to mention here that @douardda and @olasd discussed getting a short URL style domain (I think it was swh.li, but I'm not entirety sure) for various shortening needs).

This is great! Thanks a lot.

Can you also add a clean-javadoc target, equivalent of clean-images ?

• make the echo invocations a function, e.g., info()
• use a common, but distinguishable prefix instead of just "#" which will allow to search for progress info in the potentially huge log-files, e.g., "* swh-graph:"
• maybe add a step number v. total number of steps (e.g., 2/5)

Sure, let's do that, but I've never touched anything related to SWH on PyPI, so I've no idea how to make it happens.
Whoever has an idea of how do to that, please just go ahead. (And feel free to tag any recent version for PyPI publishing; the currently tagged version was completely arbitrary, just because a tag was needed for $something.)

No objection from me on adding a more abstract node type. It would be a nicer API and, given it's gonna be on the python side only, it won't have any impact on perf.

In D1768#41627, @haltode wrote:

The mapping has always been a separate command but we can indeed add it to the generate_graph.sh script.

landed in a043b0ee04aae50f8c26f6a06aac1e6c9247340a

In D1768#41565, @haltode wrote:

Note that you also need to update the java/server/src/test/dataset/generate_graph.sh script since the Docker environment has changed.

• tests: update generate_graph.sh to match new docker layout

We want to test that the client part of a complete client<->server interaction works properly. The best way to do that is, in fact, to rerun the same tests we run on the server side, but via the Python client. If there is an easy way to just reuse the same test code (e.g., by generating Python tests from the Java ones, or vice-versa), go for it. But probably it isn't worth it, as there isn't much test code anyway. If there is no way to keep parity, we should go for something minimal on the Python side, e.g., just test one call per API endpoint, and keep a more complete coverage on the Java side (again: or vice-versa, if we prefer to maintain the Python test code base than the Java one).

Tnx, will do.

No, you're right, I didn't think of the global namespace of unified documentation.

• docker doc: add --publish to the run invocation

• docker doc: drop the list of files generated by Setup class

• docker doc: further shorten CLI examples using relative paths

due to this bug (I suppose), trying to generate the various mapping files for a compressed graph that also includes swh:1:ori:... PIDs fails with:

Pre-computing node id maps...
Exception in thread "main" java.lang.IllegalArgumentException: Unknown SWH ID type in: swh:1:ori:4135fe80baeff9983f73e94b02da92f618cbb6c7
	at org.softwareheritage.graph.SwhId.<init>(SwhId.java:44)
	at org.softwareheritage.graph.backend.Setup.precomputeNodeIdMap(Setup.java:108)
	at org.softwareheritage.graph.backend.Setup.main(Setup.java:47)

Here's a first stab at an API for the py4j bindings that would be nice to use.

Instead of this, please add a pair of private methods to the Endpoint class, one to start timing (e.g., this.startTiming()), one to end it and return the diff w.r.t. the start time (e.g., this.stopTiming()) , and make all endpoint methods invoke the two methods and log the result.

I was more thinking of lifting this up to the traversal algo, ideally not having to do a test at every edge that is followed.
Done this way, this is not gonna gain you much, only a couple of array lookup per edge.

in f13a43d697eb0d10ba59a4789847742607c49aaa I've added swh-graph to the mrconfig of swh-environment, let's see what the CI has to say about that… (cc: @douardda)

In D1741#40411, @haltode wrote:

Concerning links to swh-graph/docs it needs to wait for integration with swh-docs and then add external html links.