Page MenuHomeSoftware Heritage
Feed Advanced Search

Aug 5 2019

haltode closed T1933: bad invocation of o.s.graph.backend.Setup in docker doc as Resolved by committing rDGRPH4013dbc12e43: docs: fix backend.Setup command arguments.
Aug 5 2019, 2:11 PM · Compressed graph service
zack added a project to T1943: Publish swh-graph to PyPI: Compressed graph service.

Sure, let's do that, but I've never touched anything related to SWH on PyPI, so I've no idea how to make it happen.
Whoever has an idea of how do to that, please just go ahead. (And feel free to tag any recent version for PyPI publishing; the currently tagged version was completely arbitrary, just because a tag was needed for $something.)

Aug 5 2019, 1:07 PM · Compressed graph service
haltode closed T1867: compress Merkle DAG and origin nodes together as Resolved.
Aug 5 2019, 10:24 AM · Compressed graph service
haltode added a comment to T1867: compress Merkle DAG and origin nodes together.

Due to multiple server maintenance, the process was re-started a few times, but it is now finished and results are uploaded in the annex: https://annex.softwareheritage.org/public/dataset/graph/latest/compressed/all+ori/

Aug 5 2019, 10:24 AM · Compressed graph service

Aug 3 2019

haltode added a comment to T1904: build developer documentation for swh-graph.

What was tried so far:

Aug 3 2019, 11:13 PM · Documentation, Compressed graph service
zack added a comment to T1884: python bindings for compressed graph access.

No objection from me on adding a more abstract node type. It would be a nicer API and, given it's gonna be on the python side only, it won't have any impact on perf.

Aug 3 2019, 8:30 AM · Compressed graph service

Aug 2 2019

seirl added a comment to T1884: python bindings for compressed graph access.

Sounds good. No 'node' type then, we just use IDs? Maybe a Node type would allow to do stuff like neighbors() directly on the Node instance?

Aug 2 2019, 4:21 PM · Compressed graph service

Aug 1 2019

ardumont added projects to P486 graph.tf: System administration, Compressed graph service.
Aug 1 2019, 5:41 PM · Compressed graph service, System administration
haltode closed T1920: graph service: add tests for the python client as Resolved by committing rDGRPHc57819ce34ae: client: add API endpoints integration tests.
Aug 1 2019, 2:49 PM · Compressed graph service

Jul 31 2019

haltode raised the priority of T1930: swh-graph: ship swh-graph.jar in the docker container from Low to Normal.
Jul 31 2019, 3:03 PM · Compressed graph service
haltode triaged T1941: Automatically generate mapping files after compressing graph as Normal priority.
Jul 31 2019, 3:02 PM · Compressed graph service
haltode added a revision to T1920: graph service: add tests for the python client: D1787: Client: add API endpoints integration tests.
Jul 31 2019, 2:01 PM · Compressed graph service

Jul 30 2019

haltode added a comment to T1920: graph service: add tests for the python client.

One test call per endpoint seems enough right now, the Python side relies entirely on swh custom REST API class for client <-> server interaction (which has tests on its own).

Jul 30 2019, 9:49 AM · Compressed graph service

Jul 29 2019

zack added a comment to T1920: graph service: add tests for the python client.

We want to test that the client part of a complete client<->server interaction works properly. The best way to do that is, in fact, to rerun the same tests we run on the server side, but via the Python client. If there is an easy way to just reuse the same test code (e.g., by generating Python tests from the Java ones, or vice-versa), go for it. But probably it isn't worth it, as there isn't much test code anyway. If there is no way to keep parity, we should go for something minimal on the Python side, e.g., just test one call per API endpoint, and keep a more complete coverage on the Java side (again: or vice-versa, if we prefer to maintain the Python test code base than the Java one).

Jul 29 2019, 10:03 PM · Compressed graph service
haltode closed T1939: Measure memory needs for a swh-graph Azure VM, a subtask of T1885: benchmark swh-graph use cases on the full graph, as Resolved.
Jul 29 2019, 6:40 PM · Compressed graph service
haltode closed T1939: Measure memory needs for a swh-graph Azure VM as Resolved.

Running swh-graph (with only the default graph not its transposed) requires ~125GB of RAM.

Jul 29 2019, 6:40 PM · Compressed graph service
haltode added a comment to T1920: graph service: add tests for the python client.

What kind of tests do we want for the client side code? Checking the resulting json format for each endpoints?

Jul 29 2019, 3:01 PM · Compressed graph service
haltode added a subtask for T1885: benchmark swh-graph use cases on the full graph: T1939: Measure memory needs for a swh-graph Azure VM.
Jul 29 2019, 2:12 PM · Compressed graph service
haltode added a parent task for T1939: Measure memory needs for a swh-graph Azure VM: T1885: benchmark swh-graph use cases on the full graph.
Jul 29 2019, 2:12 PM · Compressed graph service
haltode triaged T1939: Measure memory needs for a swh-graph Azure VM as High priority.
Jul 29 2019, 2:11 PM · Compressed graph service
haltode closed T1938: swh-graph: NullPointerException upon (wrong) /walk from cnt to snp as Resolved by committing rDGRPHb884ff82ee09: server: use constructor instead of null for default value.
Jul 29 2019, 1:28 PM · Compressed graph service
haltode closed T1915: Add support for origin nodes in graph service API as Resolved by committing rDGRPH93f1fb8ed6b9: server: add 'origin' node type.
Jul 29 2019, 1:15 PM · Compressed graph service
haltode closed T1915: Add support for origin nodes in graph service API, a subtask of T1867: compress Merkle DAG and origin nodes together, as Resolved.
Jul 29 2019, 1:15 PM · Compressed graph service
haltode added a revision to T1915: Add support for origin nodes in graph service API: D1781: Add 'origin' node type.
Jul 29 2019, 12:35 PM · Compressed graph service
haltode closed T1922: swh-graph optimization: bypass edge restriction checks when edges=* as Resolved by committing rDGRPHa8d4b625d917: server: bypass edge restriction checks when edges=*.
Jul 29 2019, 10:43 AM · Compressed graph service
haltode closed T1922: swh-graph optimization: bypass edge restriction checks when edges=*, a subtask of T1885: benchmark swh-graph use cases on the full graph, as Resolved.
Jul 29 2019, 10:43 AM · Compressed graph service
haltode added a comment to T1915: Add support for origin nodes in graph service API.

Yes this is related, there are safe checks when creating a new SwhId from a string form, and right now the code for the type looks like:

Jul 29 2019, 10:36 AM · Compressed graph service
haltode closed T1921: swh-graph: add logging of endpoint timing, a subtask of T1885: benchmark swh-graph use cases on the full graph, as Resolved.
Jul 29 2019, 10:09 AM · Compressed graph service
haltode closed T1921: swh-graph: add logging of endpoint timing as Resolved by committing rDGRPHebfa6310c5ab: server: add logging of endpoint timing.
Jul 29 2019, 10:09 AM · Compressed graph service
haltode claimed T1937: nicer landing page for the swh-graph REST API.
Jul 29 2019, 8:38 AM · Compressed graph service
haltode claimed T1933: bad invocation of o.s.graph.backend.Setup in docker doc.
Jul 29 2019, 8:37 AM · Compressed graph service
haltode claimed T1920: graph service: add tests for the python client.
Jul 29 2019, 8:36 AM · Compressed graph service

Jul 28 2019

zack triaged T1938: swh-graph: NullPointerException upon (wrong) /walk from cnt to snp as Normal priority.
Jul 28 2019, 7:22 PM · Compressed graph service
zack triaged T1937: nicer landing page for the swh-graph REST API as Low priority.
Jul 28 2019, 7:04 PM · Compressed graph service
zack triaged T1936: integrate swh-graph into the docker environment as Wishlist priority.
Jul 28 2019, 6:56 PM · Docker environment, Compressed graph service

Jul 25 2019

zack added a parent task for T1915: Add support for origin nodes in graph service API: T1867: compress Merkle DAG and origin nodes together.
Jul 25 2019, 4:58 PM · Compressed graph service
zack added a subtask for T1867: compress Merkle DAG and origin nodes together: T1915: Add support for origin nodes in graph service API.
Jul 25 2019, 4:58 PM · Compressed graph service
zack raised the priority of T1915: Add support for origin nodes in graph service API from Normal to High.

due to this bug (I suppose), trying to generate the various mapping files for a compressed graph that also includes swh:1:ori:... PIDs fails with:

Pre-computing node id maps...
Exception in thread "main" java.lang.IllegalArgumentException: Unknown SWH ID type in: swh:1:ori:4135fe80baeff9983f73e94b02da92f618cbb6c7
	at org.softwareheritage.graph.SwhId.<init>(SwhId.java:44)
	at org.softwareheritage.graph.backend.Setup.precomputeNodeIdMap(Setup.java:108)
	at org.softwareheritage.graph.backend.Setup.main(Setup.java:47)
Jul 25 2019, 4:57 PM · Compressed graph service
zack triaged T1933: bad invocation of o.s.graph.backend.Setup in docker doc as Low priority.
Jul 25 2019, 4:55 PM · Compressed graph service

Jul 24 2019

zack updated the task description for T1930: swh-graph: ship swh-graph.jar in the docker container.
Jul 24 2019, 7:54 AM · Compressed graph service
zack triaged T1930: swh-graph: ship swh-graph.jar in the docker container as Low priority.
Jul 24 2019, 7:51 AM · Compressed graph service

Jul 21 2019

zack updated subscribers of T1884: python bindings for compressed graph access.

Here's a first stab at an API for the py4j bindings that would be nice to use.

Jul 21 2019, 2:31 PM · Compressed graph service
zack triaged T1926: FUSE filesystem to navigate the archive as Wishlist priority.
Jul 21 2019, 2:05 PM · Software Heritage filesystem

Jul 19 2019

haltode added a revision to T1921: swh-graph: add logging of endpoint timing: D1755: Add logging of endpoint timing.
Jul 19 2019, 4:05 PM · Compressed graph service
haltode added a revision to T1922: swh-graph optimization: bypass edge restriction checks when edges=*: D1753: Bypass edge restriction checks when edges=*.
Jul 19 2019, 3:32 PM · Compressed graph service
zack updated subscribers of T1921: swh-graph: add logging of endpoint timing.
Jul 19 2019, 10:08 AM · Compressed graph service
zack updated the task description for T1922: swh-graph optimization: bypass edge restriction checks when edges=*.
Jul 19 2019, 10:08 AM · Compressed graph service
zack triaged T1922: swh-graph optimization: bypass edge restriction checks when edges=* as High priority.
Jul 19 2019, 10:06 AM · Compressed graph service
zack updated the task description for T1885: benchmark swh-graph use cases on the full graph.
Jul 19 2019, 10:06 AM · Compressed graph service
zack triaged T1921: swh-graph: add logging of endpoint timing as High priority.
Jul 19 2019, 10:05 AM · Compressed graph service

Jul 18 2019

zack triaged T1920: graph service: add tests for the python client as Normal priority.
Jul 18 2019, 4:27 PM · Compressed graph service
zack changed the status of T1851: Integrate graph-compression git repo in swh-environment from Open to Work in Progress.

in f13a43d697eb0d10ba59a4789847742607c49aaa I've added swh-graph to the mrconfig of swh-environment, let's see what the CI has to say about that… (cc: @douardda)

Jul 18 2019, 10:26 AM · Compressed graph service
zack changed the status of T1851: Integrate graph-compression git repo in swh-environment, a subtask of T1887: publish swh-graph documentation at docs.s.o, from Open to Work in Progress.
Jul 18 2019, 10:26 AM · Documentation, Compressed graph service

Jul 16 2019

zack added a comment to T1904: build developer documentation for swh-graph.

*SAD TROMBONE*.

Jul 16 2019, 4:06 PM · Documentation, Compressed graph service
haltode added a comment to T1904: build developer documentation for swh-graph.

From the sphinx-maven plugin documentation :

Jul 16 2019, 3:58 PM · Documentation, Compressed graph service
haltode closed T1901: refactor graph API to use integer IDs in the kernel and translate to/from SWH PIDs on outer layers, a subtask of T1884: python bindings for compressed graph access, as Resolved.
Jul 16 2019, 9:40 AM · Compressed graph service
haltode closed T1902: Use in-memory bitmap to store node->types relations in graph API, a subtask of T1901: refactor graph API to use integer IDs in the kernel and translate to/from SWH PIDs on outer layers, as Resolved.
Jul 16 2019, 9:40 AM · Compressed graph service
haltode closed T1898: swh-graph: refactor algo implementations to not forcibly memoize results as Resolved by committing rDGRPHd56bfa886d2e: server: separate id mapping from traversals using node2type map.
Jul 16 2019, 9:40 AM · Compressed graph service
haltode closed T1902: Use in-memory bitmap to store node->types relations in graph API as Resolved by committing rDGRPHd56bfa886d2e: server: separate id mapping from traversals using node2type map.
Jul 16 2019, 9:40 AM · Compressed graph service
haltode closed T1901: refactor graph API to use integer IDs in the kernel and translate to/from SWH PIDs on outer layers as Resolved by committing rDGRPHd56bfa886d2e: server: separate id mapping from traversals using node2type map.
Jul 16 2019, 9:40 AM · Compressed graph service

Jul 15 2019

zack edited P461 swh-graph use cases.
Jul 15 2019, 4:19 PM · Compressed graph service
haltode added a revision to T1898: swh-graph: refactor algo implementations to not forcibly memoize results: D1736: Separate id mapping from traversals using node2type map.
Jul 15 2019, 3:20 PM · Compressed graph service
haltode added a revision to T1902: Use in-memory bitmap to store node->types relations in graph API: D1736: Separate id mapping from traversals using node2type map.
Jul 15 2019, 2:42 PM · Compressed graph service
haltode added a revision to T1901: refactor graph API to use integer IDs in the kernel and translate to/from SWH PIDs on outer layers: D1736: Separate id mapping from traversals using node2type map.
Jul 15 2019, 2:42 PM · Compressed graph service

Jul 14 2019

haltode triaged T1915: Add support for origin nodes in graph service API as Normal priority.
Jul 14 2019, 7:29 PM · Compressed graph service

Jul 12 2019

grouss edited P461 swh-graph use cases.
Jul 12 2019, 2:58 PM · Compressed graph service

Jul 11 2019

haltode added a revision to T1902: Use in-memory bitmap to store node->types relations in graph API: D1726: Add node id -> node types bitmap.
Jul 11 2019, 4:31 PM · Compressed graph service
haltode closed T1903: Add graph service README files as Resolved.

Closed by 946d235ebdac.

Jul 11 2019, 4:00 PM · Compressed graph service
haltode closed T1903: Add graph service README files, a subtask of T1851: Integrate graph-compression git repo in swh-environment, as Resolved.
Jul 11 2019, 4:00 PM · Compressed graph service
haltode added a comment to T1898: swh-graph: refactor algo implementations to not forcibly memoize results.

More work needs to be done after D1700 (integrate with node->type map, better isolation for swh id/longs, etc.), so I'm leaving this open.

Jul 11 2019, 3:44 PM · Compressed graph service
haltode added a revision to T1903: Add graph service README files: D1721: Add swh-graph server side README.
Jul 11 2019, 3:39 PM · Compressed graph service
haltode claimed T1888: graph API documentation: clarify the relationship between directory=backward and edges=.
Jul 11 2019, 1:33 PM · Documentation, Compressed graph service
haltode claimed T1898: swh-graph: refactor algo implementations to not forcibly memoize results.
Jul 11 2019, 1:33 PM · Compressed graph service

Jul 10 2019

vlorentz reopened T1731: Intrinsic identifiers for origins, a subtask of T1867: compress Merkle DAG and origin nodes together, as Open.
Jul 10 2019, 4:59 PM · Compressed graph service
vlorentz closed T1731: Intrinsic identifiers for origins, a subtask of T1867: compress Merkle DAG and origin nodes together, as Resolved.
Jul 10 2019, 4:59 PM · Compressed graph service

Jul 9 2019

zack added a parent task for T1904: build developer documentation for swh-graph: T1887: publish swh-graph documentation at docs.s.o.
Jul 9 2019, 5:03 PM · Documentation, Compressed graph service
zack added a subtask for T1887: publish swh-graph documentation at docs.s.o: T1904: build developer documentation for swh-graph.
Jul 9 2019, 5:03 PM · Documentation, Compressed graph service
zack removed a parent task for T1904: build developer documentation for swh-graph: T1851: Integrate graph-compression git repo in swh-environment.
Jul 9 2019, 5:02 PM · Documentation, Compressed graph service
zack removed a subtask for T1851: Integrate graph-compression git repo in swh-environment: T1904: build developer documentation for swh-graph.
Jul 9 2019, 5:02 PM · Compressed graph service
zack triaged T1904: build developer documentation for swh-graph as Normal priority.
Jul 9 2019, 5:02 PM · Documentation, Compressed graph service
haltode added a subtask for T1851: Integrate graph-compression git repo in swh-environment: T1903: Add graph service README files.
Jul 9 2019, 4:10 PM · Compressed graph service
haltode added a parent task for T1903: Add graph service README files: T1851: Integrate graph-compression git repo in swh-environment.
Jul 9 2019, 4:10 PM · Compressed graph service
haltode triaged T1903: Add graph service README files as High priority.
Jul 9 2019, 4:10 PM · Compressed graph service
haltode claimed T1901: refactor graph API to use integer IDs in the kernel and translate to/from SWH PIDs on outer layers.
Jul 9 2019, 3:01 PM · Compressed graph service
haltode added a subtask for T1901: refactor graph API to use integer IDs in the kernel and translate to/from SWH PIDs on outer layers: T1902: Use in-memory bitmap to store node->types relations in graph API.
Jul 9 2019, 3:00 PM · Compressed graph service
haltode added a parent task for T1902: Use in-memory bitmap to store node->types relations in graph API: T1901: refactor graph API to use integer IDs in the kernel and translate to/from SWH PIDs on outer layers.
Jul 9 2019, 3:00 PM · Compressed graph service
haltode triaged T1902: Use in-memory bitmap to store node->types relations in graph API as High priority.
Jul 9 2019, 2:59 PM · Compressed graph service
zack added a parent task for T1901: refactor graph API to use integer IDs in the kernel and translate to/from SWH PIDs on outer layers: T1884: python bindings for compressed graph access.
Jul 9 2019, 2:55 PM · Compressed graph service
zack added a subtask for T1884: python bindings for compressed graph access: T1901: refactor graph API to use integer IDs in the kernel and translate to/from SWH PIDs on outer layers.
Jul 9 2019, 2:55 PM · Compressed graph service
zack triaged T1901: refactor graph API to use integer IDs in the kernel and translate to/from SWH PIDs on outer layers as Normal priority.
Jul 9 2019, 2:54 PM · Compressed graph service
zack changed the status of T1867: compress Merkle DAG and origin nodes together from Open to Work in Progress.

this has been started on sexus yesterday, ETA: next monday-ish

Jul 9 2019, 2:52 PM · Compressed graph service
haltode closed T1889: graph API: add endpoint to return the leaves of a subgraph as Resolved.

Done in D1699.

Jul 9 2019, 8:59 AM · Compressed graph service
haltode closed T1886: graph API: add endpoint to return the adjacency list of a node as Resolved.

Done in D1698.

Jul 9 2019, 8:46 AM · Compressed graph service
zack triaged T1898: swh-graph: refactor algo implementations to not forcibly memoize results as Low priority.
Jul 9 2019, 8:24 AM · Compressed graph service

Jul 7 2019

zack edited P461 swh-graph use cases.
Jul 7 2019, 8:47 PM · Compressed graph service
zack updated the task description for T1889: graph API: add endpoint to return the leaves of a subgraph.
Jul 7 2019, 8:44 PM · Compressed graph service
zack updated the task description for T1886: graph API: add endpoint to return the adjacency list of a node.
Jul 7 2019, 8:43 PM · Compressed graph service
zack triaged T1889: graph API: add endpoint to return the leaves of a subgraph as Normal priority.
Jul 7 2019, 8:42 PM · Compressed graph service
zack created T1889: graph API: add endpoint to return the leaves of a subgraph.
Jul 7 2019, 8:42 PM · Compressed graph service
zack renamed T1886: graph API: add endpoint to return the adjacency list of a node from Graph API: add endpoint to return the adjacency list of a node to graph API: add endpoint to return the adjacency list of a node.
Jul 7 2019, 8:36 PM · Compressed graph service