In T4549#91850, @vlorentz wrote:Hmm, strangely, file/dir names are missing from the response even when omitting the mask when querying the graph server on granet; but based on the .proto file, they should be available via the successor field of Node.
Feed Advanced Search
Sep 29 2022
Sep 29 2022
Sep 26 2022
Sep 26 2022
rdicosmo updated subscribers of T4549: Write a script to generate qualified SWHID from swh-graph.
vlorentz added a comment to T4549: Write a script to generate qualified SWHID from swh-graph.
Hmm, strangely, file/dir names are missing from the response even when omitting the mask when querying the graph server on granet; but based on the .proto file, they should be available via the successor field of Node.
Sep 22 2022
Sep 22 2022
rdicosmo added a comment to T4549: Write a script to generate qualified SWHID from swh-graph.
Here is a dump of my design notes on this task.
zack added a comment to T4549: Write a script to generate qualified SWHID from swh-graph.
zack renamed T4549: Write a script to generate qualified SWHID from swh-graph from Writa a script to generate qualified SWHID from swh-graph to Write a script to generate qualified SWHID from swh-graph.
Sep 12 2022
Sep 12 2022
vlorentz reassigned T4479: uncouple the java grpc server from the python HTTP server from vlorentz to douardda.
(reassigning to @douardda as he is already working on it)
Sep 9 2022
Sep 9 2022
vlorentz removed a project from T4522: graph gRPC API: Add support for limiting traversals by number of results: Easy hack.
vlorentz changed the status of T4522: graph gRPC API: Add support for limiting traversals by number of results, a subtask of T3626: graph API: add ?limit parameter to /leaves endpoint, from Open to Work in Progress.
vlorentz changed the status of T4522: graph gRPC API: Add support for limiting traversals by number of results from Open to Work in Progress.
vlorentz triaged T4522: graph gRPC API: Add support for limiting traversals by number of results as High priority.
vlorentz changed the status of T3626: graph API: add ?limit parameter to /leaves endpoint from Open to Work in Progress.
vlorentz moved T4471: swh-graph Add java process port monitoring from Backlog to Deployed on the Compressed graph service board.
vlorentz moved T4509: [swh-graph] Configure the max_memory to use from Backlog to Deployed on the Compressed graph service board.
vlorentz triaged T4521: Non-deterministic failures in test_visit_edges_diamond_pattern[remote] as High priority.
vlorentz added a parent task for T4479: uncouple the java grpc server from the python HTTP server: T4516: swh-graph: Add jvm monitoring.
Sep 8 2022
Sep 8 2022
vsellier closed T4509: [swh-graph] Configure the max_memory to use, a subtask of T4507: Out of memory on granet, as Resolved.
diff landed and deployed, graph restarted
vsellier added a comment to T4509: [swh-graph] Configure the max_memory to use.
I forgot to mention, it seems expect during some peak, the used memory is around 350Go
vlorentz added a comment to T4507: Out of memory on granet.
Nope, I can't lower it.
vsellier changed the status of T4509: [swh-graph] Configure the max_memory to use, a subtask of T4507: Out of memory on granet, from Open to Work in Progress.
vsellier changed the status of T4509: [swh-graph] Configure the max_memory to use from Open to Work in Progress.
vlorentz added a comment to T4507: Out of memory on granet.
I'll try reducing -Xmx again...
vsellier added a comment to T4507: Out of memory on granet.
@vlorentz I assigned the task to you because if I'm not wrong you are running some experiments on granet.
I don't know what, but you should be more gentle with the server
Sep 6 2022
Sep 6 2022
Bumping this up, because I am massively using this endpoint to regenerate the license dataset; especially because I use resolve_origins=1 too
vlorentz moved T4472: swh-graph: Allow to specify the rpc port from Backlog to Wontfix / Invalid on the Compressed graph service board.
vsellier updated the task description for T4479: uncouple the java grpc server from the python HTTP server.
yes even better
douardda added a comment to T4472: swh-graph: Allow to specify the rpc port.
IMHO we should get rid of the java app being started from python (but for testing purpose).
Sep 1 2022
Sep 1 2022
Aug 30 2022
Aug 30 2022
vsellier edited projects for T4472: swh-graph: Allow to specify the rpc port, added: Compressed graph service; removed GraphQL API.
vsellier edited projects for T4471: swh-graph Add java process port monitoring , added: Compressed graph service; removed GraphQL API.
Aug 29 2022
Aug 29 2022
vlorentz moved T4468: graph dataset: redirect from annex page to doc from Backlog to Deployed on the Compressed graph service board.
done
Aug 26 2022
Aug 26 2022
vlorentz changed the status of T4422: Graph property access is not thread-safe from Open to Work in Progress.
vlorentz moved T3301: graph: add test for the "algo" parameter of walk() from Deployed to Wontfix / Invalid on the Compressed graph service board.
vlorentz changed the status of T3301: graph: add test for the "algo" parameter of walk() from Resolved to Wontfix.
vlorentz changed the status of T2103: (Debian) package py4j, a subtask of T2100: Bootstrap Debian packaging for swh.graph, from Resolved to Wontfix.
vlorentz moved T2103: (Debian) package py4j from Deployed to Wontfix / Invalid on the Compressed graph service board.
vlorentz moved T3168: Proper deployment of swh-graph with debian package from Deployed to Wontfix / Invalid on the Compressed graph service board.
vlorentz changed the status of T3168: Proper deployment of swh-graph with debian package from Resolved to Wontfix.
vlorentz changed the status of T4118: Method `visit_edges` from `NaiveClient` seems to be missing some results from Resolved to Invalid.
vlorentz changed the status of T3301: graph: add test for the "algo" parameter of walk() from Wontfix to Resolved.
vlorentz moved T2081: swh-graph: "Cannot open client FIFO" when answering HEAD requests from Backlog to Deployed on the Compressed graph service board.
vlorentz moved T3793: Add copyright notices to all swh-graph Java files from Backlog to Deployed on the Compressed graph service board.
vlorentz changed the status of T2100: Bootstrap Debian packaging for swh.graph, a subtask of T3168: Proper deployment of swh-graph with debian package, from Wontfix to Resolved.
vlorentz changed the status of T2100: Bootstrap Debian packaging for swh.graph from Wontfix to Resolved.
vlorentz changed the status of T2103: (Debian) package py4j, a subtask of T2100: Bootstrap Debian packaging for swh.graph, from Wontfix to Resolved.
vlorentz moved T3855: Document the architecture of the Java code in swh-graph from Backlog to Deployed on the Compressed graph service board.
vlorentz moved T4113: Review border case of empty response for `visit_nodes` from Backlog to Deployed on the Compressed graph service board.
vlorentz moved T4115: Some unknown SWHID errors crash the graph server from Backlog to Deployed on the Compressed graph service board.
vlorentz moved T3623: Run swh-graph with gunicorn to support multiple/parallel requests from Backlog to Deployed on the Compressed graph service board.
vlorentz changed the status of T3168: Proper deployment of swh-graph with debian package from Wontfix to Resolved.
vlorentz changed the status of T4118: Method `visit_edges` from `NaiveClient` seems to be missing some results from Invalid to Resolved.
vlorentz moved T4316: Push of swh-graph to pypi is broken from Backlog to Deployed on the Compressed graph service board.
Aug 4 2022
Aug 4 2022
seirl added a comment to T4422: Graph property access is not thread-safe.
Partially solved by https://forge.softwareheritage.org/D8191
seirl added a comment to T4422: Graph property access is not thread-safe.
Opened https://github.com/vigna/dsiutils/pull/6 as a potential fix for https://github.com/vigna/dsiutils/issues/5
seirl updated the task description for T4422: Graph property access is not thread-safe.
vlorentz added a project to T4422: Graph property access is not thread-safe: Compressed graph service.
Aug 3 2022
Aug 3 2022
Jun 30 2022
Jun 30 2022
seirl added a comment to T3259: Gracefully handle a client closing the connection in the middle of a response being streamed.
GRPC update: cancelling a GRPC stream works fine, but it doesn't seem like aiohttp is doing that when the HTTP stream is closed.
We requested a larger quota here: https://github.com/pypa/pypi-support/issues/1998
seirl closed T2100: Bootstrap Debian packaging for swh.graph, a subtask of T3168: Proper deployment of swh-graph with debian package, as Wontfix.
We are migrating away from Debian packages as a deployment tool, closing this as WONTFIX.
We are migrating away from Debian packages as a deployment tool, closing this as WONTFIX.
No longer happens after the GRPC migration:
seirl closed T2103: (Debian) package py4j, a subtask of T2100: Bootstrap Debian packaging for swh.graph, as Wontfix.
We removed the Py4J dependency by migrating to GRPC.
We no longer support multiple algorithms for shortest path requests.
Obsoleted by the migration to GRPC. Now we use GRPC's threading model, with a threadpool configurable by passing --threads to the Java service. By default, nproc is used.
Obsoleted by the migration to GRPC. We no longer create iterators on the decoded stream of a UNIX pipeline, we directly use GRPC stream iterators.
Fixed by the migration to GRPC.
Fixed in D8050
Jun 29 2022
Jun 29 2022
Jun 28 2022
Jun 28 2022
I will be solved by D7890
Jun 24 2022
Jun 24 2022
vsellier added a comment to T4340: swh-graph timeouts.
It's confirmed that the issue seems to be on the python part of the current implementation so I'm eager to see D7890 landed ;)