Hmm actually this might be harder than just using gunicorn, because the java subprocess needs to be shared between workers, hmm...

Puppet was stuck on this machine.
It's now unstuck as well.

It is now somewhat documented here: https://forge.softwareheritage.org/source/swh-environment/browse/master/docker/services/swh-graph/entrypoint.sh

Plus that documents it a bit in an automatic manner, so *thumbs up*.

I decided to make the swh-graph container create the compressed graph itself before starting. That's the easiest way to use it AND to implement it IMO.

Sure! My apologies @Hakimb, but it's thank to your work that we have realized what was the right fate for this task.

I just want to write something here that maybe isn't clear from the initial task description. This filtering must happen *after* the visit, not during. We can already change *how* the graph is visited using the edges parameter, the goal of this task is to filter the result post-visit.

Right, I suppose we can close the task then?

@seirl, @vlorentz: I see your point, and I agree. We should never have used /nested/paths for this API.
Maybe we should just reconsider this and, one @Hakimb is ready with a new traversal language proposal, we can map it to a better REST API that uses query parameters, and deal properly with 4xx return codes.

In T2981#63164, @Hakimb wrote:

questions:

1/ So for the "filter that applies to visits that return nodes one by one" part, we are talking about: neighbors, walk, visit/nodes only?

1/ So for the "filter that applies to visits that return nodes one by one" part, we are talking about: neighbors, walk, visit/nodes only?
2/ the filter is a query parameter I guess?

@zack We talked about this on IRC with @vlorentz, I think this issue is invalid. We chose to have the source and destination nodes as part of the URI in the API. Semantically, it makes sense that accessing the path without these path fragments would return a 404: it's not a missing argument but an invalid path. If we had a ?src= and a &dst= arguments instead, then having a 400 error would make sense, but in our case the semantics are really weird.

ok, so @Hakimb: go for no default value. If the query param is not passed, the visit will not stop before the end. If it's given, it will stop once the limit is reached. Call the query param ?max_edges. You will find that the java code already keeps track of the number of edges traversed, so you should just need to compare with that.

should there be a default value for it or not? (We want this to be consistent with swh-storage

To complement what @vlorentz mentioned, we should actually stop the visit after the maximum number of edges has been reached, because it is keep doing the visit (no matter how many results are returned after it) that can DoS the swh-graph backend.

1. I don't think you need to reproduce a DoS, just make sure we don't return more results than we should
2. yes, a query parameter would be good

I have two questions to make sure I understand well :

reopening, as ideally we'd like to have run the entire ORC export once to completion before closing

The ORC exporter is done, and it's likely that we won't provide CSV exports in the future, or we'll generate them from the ORC format.

To be consistent with swh-storage, swh-graph should take a limit as query parameter, but not have a hardcoded upper bound for that limit. Instead, swh-web provides that upper bound.

Now that this is (optionally) done by swh-web, I don't think we want to implement it in swh-graph too.

ohh allright.

Dear @Kaustuv942, sure, patches welcome. We do not use task claiming for non regular contributors though, just submit a patch when you have one.

Hello @zack I want to complete this task.

The fix is now deployed and proxied graph responses are now properly streamed \o/