Page MenuHomeSoftware Heritage
Feed Advanced Search

Dec 6 2021

zack accepted D4821: Add LLP compression to the WebGraph pipeline.

Just to be sure: test_pipeline() from test_cli.py is now run with all new passes as well, and as such it also testes the LLP step(s), correct?
It seems that way to me because test_pipeline() seems to be running all passes, but I'd like this to be double-checked before landing.

Dec 6 2021, 6:08 PM

Dec 4 2021

zack committed rMSLDfc9bffe30c07: check-in slides for tech presentation at #swh5years sponsors meeting (authored by zack).
check-in slides for tech presentation at #swh5years sponsors meeting
Dec 4 2021, 10:22 AM

Dec 1 2021

zack moved T2595: Add a default configuration based on graph size (eg: batch_size) from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 5:00 PM · Compressed graph service
zack changed the status of T2113: swh-graph: add support to optionally resolve ori PIDs to origin URLs from Wontfix to Resolved.
Dec 1 2021, 5:00 PM · Compressed graph service
zack moved T2112: make "swh graph map lookup" accept lists of identifiers from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 5:00 PM · Compressed graph service
zack moved T2096: CNAME for graph service: graph.internal.softwareheritage.org (?) from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 5:00 PM · Compressed graph service, System administration
zack moved T2083: provide systemd service file for swh-graph from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 5:00 PM · Compressed graph service
zack changed the status of T2056: fix swh-graph sphinx table of content from Invalid to Resolved.
Dec 1 2021, 5:00 PM · Documentation, Compressed graph service
zack moved T1933: bad invocation of o.s.graph.backend.Setup in docker doc from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 5:00 PM · Compressed graph service
zack moved T1898: swh-graph: refactor algo implementations to not forcibly memoize results from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 5:00 PM · Compressed graph service
zack moved T1888: graph API documentation: clarify the relationship between directory=backward and edges= from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 5:00 PM · Documentation, Compressed graph service
zack moved T1884: python bindings for compressed graph access from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 5:00 PM · Compressed graph service
zack moved T1937: nicer landing page for the swh-graph REST API from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 5:00 PM · Compressed graph service
zack changed the status of T1968: existing graph endpoints should not return 404 upon missing arguments from Invalid to Resolved.
Dec 1 2021, 5:00 PM · Easy hack, Compressed graph service
zack moved T1936: integrate swh-graph into the docker environment from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:38 PM · Docker environment, Compressed graph service
zack changed the status of T1930: swh-graph: ship swh-graph.jar in the docker container from Wontfix to Resolved.
Dec 1 2021, 4:38 PM · Compressed graph service
zack moved T1887: publish swh-graph documentation at docs.s.o from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:38 PM · Documentation, Compressed graph service
zack moved T1851: Integrate graph-compression git repo in swh-environment from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:38 PM · Compressed graph service
zack moved T1877: Add contextual info to compression pipeline from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T1878: Write documentation on compression process from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T1879: Write documentation on compression Docker env from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T1889: graph API: add endpoint to return the leaves of a subgraph from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T1886: graph API: add endpoint to return the adjacency list of a node from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T1901: refactor graph API to use integer IDs in the kernel and translate to/from SWH PIDs on outer layers from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T1904: build developer documentation for swh-graph from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Documentation, Compressed graph service
zack moved T1920: graph service: add tests for the python client from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T1938: swh-graph: NullPointerException upon (wrong) /walk from cnt to snp from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T1945: Return timings instead of simply logging them from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T1952: Log raw datapoint in graph benchmarks from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T2053: support graph export for the cassandra backend from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service, Storage manager
zack moved T2072: common configuration file for swh graph rpc-serve, compress, … from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T2077: add random walk endpoint with limited retries from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T2084: swh-graph: add /last endpoint variants to the REST API from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Compressed graph service
zack moved T2114: swh-graph API: add ?limit=N method variants to return first N results from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:37 PM · Easy hack, Compressed graph service
zack moved T2589: expose swh-graph API at archive.s.o/api/1/graph/ from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · System administration, Web app, Compressed graph service
zack moved T2900: Public graph/ API does not handle streaming results from endpoints from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · System administration, Compressed graph service, Web app
zack moved T1862: Implement new graph API specifications from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · Compressed graph service
zack moved T1902: Use in-memory bitmap to store node->types relations in graph API from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · Compressed graph service
zack moved T1903: Add graph service README files from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · Compressed graph service
zack moved T1915: Add support for origin nodes in graph service API from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · Compressed graph service
zack moved T1867: compress Merkle DAG and origin nodes together from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · Compressed graph service
zack moved T1921: swh-graph: add logging of endpoint timing from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · Compressed graph service
zack moved T1922: swh-graph optimization: bypass edge restriction checks when edges=* from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · Compressed graph service
zack moved T1939: Measure memory needs for a swh-graph Azure VM from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · Compressed graph service
zack moved T1941: Automatically generate mapping files after compressing graph from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · Compressed graph service
zack moved T1944: use a compact, binary format for node ids mapping files from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:36 PM · Compressed graph service
zack moved T1950: Reduce RAM usage for generating mapping files from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:35 PM · Compressed graph service
zack moved T2045: add support for reverse lookup from swh:1:ori:... PIDs to origin URLs from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:35 PM · Compressed graph service, Storage manager
zack moved T1951: Reduce RAM usage in graph API endpoints from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:35 PM · Compressed graph service
zack moved T1885: benchmark swh-graph use cases on the full graph from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:35 PM · Compressed graph service
zack moved T2054: CI: ImportMismatchError when running on swh-graph from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:35 PM · Compressed graph service, Continuous Integration
zack moved T2055: swh-graph CI hangs badly when py4j doesn't find needed files from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:35 PM · Continuous Integration, Compressed graph service
zack moved T1868: refresh compressed representation of the archive from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:35 PM · Compressed graph service
zack moved T2492: swh-graph: loading maps fail when available memory is too low: Cannot allocate memory from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:35 PM · Compressed graph service
zack moved T2530: Write a simple "quick start" for swh-graph from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:35 PM · Documentation, Compressed graph service
zack moved T2642: swh-graph: fix CI from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:34 PM · Compressed graph service
zack moved T2768: unbreak swh-graph CI from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:34 PM · Continuous Integration, Compressed graph service
zack moved T3624: Update swh-graph from 0.3.0 to 0.5.0 on granet from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:34 PM · Compressed graph service, System administration
zack moved T3564: Puppetize graph service and add icinga alert from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:34 PM · System administration, Compressed graph service, Puppet recipes
zack raised the priority of T2647: add LLP support to graph compression pipeline from Normal to High.
Dec 1 2021, 4:27 PM · Compressed graph service
zack moved T1967: REST server hangs when loading entire graph from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:26 PM · Compressed graph service
zack moved T1943: Publish swh-graph to PyPI from Backlog to Deployed on the Compressed graph service board.
Dec 1 2021, 4:26 PM · Compressed graph service
zack moved T3740: swh-graph: Translate node IDs on the Java side, not Python side from In progress to Implemented on the Compressed graph service board.
Dec 1 2021, 4:26 PM · Compressed graph service
zack raised the priority of T2400: Ingest current and historical Ubuntu releases from Normal to High.
Dec 1 2021, 12:39 PM · System administration, Debian loader, Package Loader, Archive coverage

Nov 29 2021

zack changed the visibility for F5047255: sched.tar.gz.
Nov 29 2021, 1:16 PM
zack added a comment to T3755: misleading 100% known summary in sunburst rendering.

I've tried replacing the content of foo.txt with something unknown to the archive (random garbage) and the sunburst rendering still shows 100.0%.
So it could also be a rounding error instead.
Either way, it is misleading and should be fixed.

Nov 29 2021, 1:14 PM · Code scanner
zack triaged T3755: misleading 100% known summary in sunburst rendering as Low priority.
Nov 29 2021, 1:10 PM · Code scanner
zack triaged T3754: scanning sunburst rendering fail with "ValueError: Empty data passed with indices specified." as Normal priority.
Nov 29 2021, 1:04 PM · Code scanner
zack created T3754: scanning sunburst rendering fail with "ValueError: Empty data passed with indices specified.".
Nov 29 2021, 1:04 PM · Code scanner

Nov 23 2021

zack changed the status of T2647: add LLP support to graph compression pipeline from Open to Work in Progress.
Nov 23 2021, 1:47 PM · Compressed graph service
zack changed the status of T3161: graph service: add anti-DoS limit on the number of edges traversed, a subtask of T2220: swh-graph in production, from Open to Work in Progress.
Nov 23 2021, 1:38 PM · Roadmap 2022, meta-task, Roadmap 2021, Compressed graph service
zack changed the status of T3161: graph service: add anti-DoS limit on the number of edges traversed from Open to Work in Progress.
Nov 23 2021, 1:38 PM · Compressed graph service
zack changed the status of T3739: swh-graph: Remove SWHID -> Node ID mapping, use MPH instead from Open to Work in Progress.
Nov 23 2021, 1:36 PM · Compressed graph service
zack changed the status of T3740: swh-graph: Translate node IDs on the Java side, not Python side from Open to Work in Progress.
Nov 23 2021, 1:36 PM · Compressed graph service
zack changed the status of T3740: swh-graph: Translate node IDs on the Java side, not Python side, a subtask of T3739: swh-graph: Remove SWHID -> Node ID mapping, use MPH instead, from Open to Work in Progress.
Nov 23 2021, 1:36 PM · Compressed graph service

Nov 18 2021

zack accepted D6656: docs/index: Group swh.* entries under a top level "API reference" entry.

<3

Nov 18 2021, 2:15 PM
zack triaged T3737: documentation sidebar: group swh.* module doc under a "API reference" heading as Low priority.
Nov 18 2021, 10:54 AM · Documentation

Nov 15 2021

zack closed T3729: stuck vault cooking tasks as Resolved.

the cooking has completed now

Nov 15 2021, 11:36 AM · Vault, System administration
zack triaged T3729: stuck vault cooking tasks as Normal priority.
Nov 15 2021, 10:50 AM · Vault, System administration

Nov 11 2021

zack committed rMSLDe26c195f491d: check in slides for SFScon 2021 talk (authored by zack).
check in slides for SFScon 2021 talk
Nov 11 2021, 4:02 PM
zack committed rMSLDf100c34864bf: status module: update total size figure (authored by zack).
status module: update total size figure
Nov 11 2021, 4:02 PM

Nov 9 2021

zack added a comment to T1538: Add "forge" now.

Is the request for regular pulling?
If yes, it should be added for visibility on the home page of the archive.
If no, it should be clear this is a one shot thing.

Nov 9 2021, 9:48 AM · Add Forge Now , Roadmap 2022, meta-task, Roadmap 2021

Nov 8 2021

zack updated the task description for T3717: Ingest opam instance https://coq.inria.fr/opam/released/.
Nov 8 2021, 2:41 PM · System administration, Archive coverage, Opam
zack added a project to T3717: Ingest opam instance https://coq.inria.fr/opam/released/: Archive coverage.
Nov 8 2021, 2:41 PM · System administration, Archive coverage, Opam

Nov 3 2021

zack added a comment to T3621: Create a production read-only objstorage.

Where is the documentation on how to access the new read-only object storage?
(hint hint :-))

Nov 3 2021, 6:00 PM · System administration

Nov 2 2021

zack added a reviewer for D6594: Add parameter to load a single graph direction in memory.: seirl.
Nov 2 2021, 1:54 PM

Oct 31 2021

zack added a comment to T2983: graph service: allow loading in memory only one direction of the graph.

Nope, if loaded with only one direction, traversals will only be possible in the loaded direction.
This will essentially be a trade-off setting for people who cannot (or doesn't want to) load both direction.
It is fine to fail (gracefully, with an error) traversals requested in a direction that corresponds to a non-loaded graph.

Oct 31 2021, 7:10 PM · Compressed graph service
zack added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

Thanks for the summaries @olasd, both here and on list.
I've followed up on list.

Oct 31 2021, 4:11 PM · Git loader

Oct 28 2021

zack closed D6567: FAQ: point to developer setup + minor fixes.
Oct 28 2021, 10:40 AM
zack committed rDDOC7e3d259a1598: FAQ: point to developer setup + minor fixes (authored by zack).
FAQ: point to developer setup + minor fixes
Oct 28 2021, 10:40 AM
zack updated the diff for D6567: FAQ: point to developer setup + minor fixes.

rebase

Oct 28 2021, 10:24 AM

Oct 27 2021

zack requested review of D6567: FAQ: point to developer setup + minor fixes.
Oct 27 2021, 4:58 PM

Oct 26 2021

zack changed the edit policy for T2845: Improve Subversion loader and develop CVS loader.
Oct 26 2021, 11:44 AM · Archive coverage

Oct 22 2021

zack added a comment to T3621: Create a production read-only objstorage.

@vsellier sure, and thanks! "Basic auth" is in the HTTP sense, right? So username/password pairs that we can add on demand, correct?

Oct 22 2021, 6:12 PM · System administration

Oct 19 2021

zack committed rMSLD05f893596a65: talk IDIA: last bits (authored by zack).
talk IDIA: last bits
Oct 19 2021, 3:12 PM
zack committed rMSLDcc87f86babff: academic adoption module: minor style and typo fixes (authored by zack).
academic adoption module: minor style and typo fixes
Oct 19 2021, 10:44 AM
zack committed rMSLD94c77a7be1a7: check in slides for talk at Telecom/IDIA PFDay (authored by zack).
check in slides for talk at Telecom/IDIA PFDay
Oct 19 2021, 10:44 AM

Oct 18 2021

zack accepted D6493: fuse: docs/images/Makefile: Drop unused pdf generation from the build.
Oct 18 2021, 12:01 PM
zack requested changes to D6493: fuse: docs/images/Makefile: Drop unused pdf generation from the build.
Oct 18 2021, 11:40 AM

Oct 15 2021

zack added a comment to D6479: bib/install: disable pip self ugprade.

There is a typo in commit message: s/bib/bin/.

Oct 15 2021, 11:50 AM