- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Jun 29 2022
Jun 29 2022
Update webgraph-big to 3.7.0
seirl committed rDGRPH53b7d553ba06: GraphServer: Use nproc threads in threadpool by default (authored by seirl).
GraphServer: Use nproc threads in threadpool by default
- SwhBidirectionalGraph: fix copy() not actually copying subgraphs
- GetNode: make endpoint thread-safe with a lightweight copy of the graph
- GraphServer: Use nproc threads in threadpool by default
- Update webgraph-big to 3.7.0
Jun 28 2022
Jun 28 2022
seirl committed rSPSITEf64bcc100756: swhgraphshm: do not hardcode the graph path, use /latest symlink (authored by seirl).
swhgraphshm: do not hardcode the graph path, use /latest symlink
Rebase
Jun 24 2022
Jun 24 2022
- Remove old useless classes, including Traversal
- Regenerate python protobuf documentation
- style: typos and indent fixes
Jun 23 2022
Jun 23 2022
seirl added a revision to T2103: (Debian) package py4j: D7890: Migrate low-level RPC API from Py4J to GRPC.
seirl added a revision to T4340: swh-graph timeouts: D7890: Migrate low-level RPC API from Py4J to GRPC.
seirl added tasks to D7890: Migrate low-level RPC API from Py4J to GRPC: Unknown Object (Maniphest Task), T4340: swh-graph timeouts, T2103: (Debian) package py4j, T3259: Gracefully handle a client closing the connection in the middle of a response being streamed.
- Add inline docstrings to Java GRPC server
Will be replaced by the GRPC API
- doc: First full draft of GRPC API documentation
- SwhGraphProperties: remove useless IOExceptions
- SwhUnidirectionalGraph: make constructor with properties public
- GRPC: initial commit with protobuf + java server
- Add CheckSwhid() and Stats() RPC methods
- RPC: temporarily use loadLabelled() because loadLabelledMapped() doesn't work yet
- Java: add proto/ dir symlink
- Python: migrate HTTP API to GRPC
- Java tests: migrate to GRPC
- Remove old Java HTTP server, replaced by GRPC
- NodeIdMap: use new *MappedBigList classes, remove deprecated classes and MapFile
- Remove now useless backend.py
- Remove now useless dot.py
- Reorganize Python files (RPC/HTTP server distinction)
- flake: exclude swh/graph/rpc dir
- requirements-test: add grpc-stubs
- mypy.ini: ignore grpc generated files
- Add rpc.StatsTest
- proto: migrate to FieldMask to filter out fields
- Add FindPathTo and FindPathBetween endpoints
- Remove GraphDirection.BOTH (labelled iteration not supported yet)
- Traversal: test traversals from multiple sources/to multiple dests
- Traversal: test impossible paths
- Traversal: check for invalid arguments
- Traversal: simplify the StopTraversal logic
- Remove CheckSwhid, use GetNode instead
- Add tests for CountNodes/CountEdges
- Traversal: add max depth and common ancestors tests
- Traversal: add max edge tests
- Document protobuf/grpc services and fields
- More protobuf/grpc documentation, better field names
- swhgraph.proto: small documentation fixes
- doc: add GRPC page skeleton
Jun 14 2022
Jun 14 2022
Still needs some work/documentation
rebase
ORC: handle nullable columns properly
Jun 9 2022
Jun 9 2022
python cli: fix compress documentation
(4. doesn't work because libhadoop.so isn't packaged in maven)
Jun 1 2022
Jun 1 2022
seirl committed rDGRPHf5e658596c8f: SwhUnidirectionalGraph: fix copy() for labelled graphs (authored by seirl).
SwhUnidirectionalGraph: fix copy() for labelled graphs
seirl planned changes to D7936: [WIP] collabgraph: add tool to generate author collaboration graphs.
seirl retitled D7936: [WIP] collabgraph: add tool to generate author collaboration graphs from collabgraph: add tool to generate author collaboration graphs to [WIP] collabgraph: add tool to generate author collaboration graphs.
seirl added a comment to D7936: [WIP] collabgraph: add tool to generate author collaboration graphs.
This was meant to be a draft, but I couldn't find the button to make it so
seirl requested review of D7936: [WIP] collabgraph: add tool to generate author collaboration graphs.
May 23 2022
May 23 2022
seirl committed rDDATASET5916d8b5fc97: docs: add missing refs, comment outdated schema (authored by seirl).
docs: add missing refs, comment outdated schema
May 20 2022
May 20 2022
Fixed in D7839
Documentation overhaul
Review fixes
May 19 2022
May 19 2022
Review fixes
Thanks for all the reviews, I made a first pass with the easiest fixes.
May 17 2022
May 17 2022
fix
typo
Review fixes
May 16 2022
May 16 2022
May 14 2022
May 14 2022
seirl committed rDGRPH579f5a9e89be: requirements-test.txt: add missing pytest-asyncio (authored by seirl).
requirements-test.txt: add missing pytest-asyncio
seirl committed rDGRPH768090281f75: config: fix logging warning (warn() -> warning()) (authored by seirl).
config: fix logging warning (warn() -> warning())
cli: fix compress docstring
Remove dead code from the Python interface
May 12 2022
May 12 2022
In D7814#203336, @zack wrote:I'm fine with this code cleanup, with one caveat: that we document/ship the systemd startup service (and its meaning, including some intuitions about the trade-offs you mention) somewhere, in replacement of the cachemount command.
May 11 2022
May 11 2022
In D7814#203309, @ardumont wrote:@seirl: So, for future version deployments, nothing changes, right?
May 10 2022
May 10 2022
Test dataset: recompress with new pipeline
seirl committed rDGRPH53f9fc5c413c: Test dataset: update with new ORC naming scheme (authored by seirl).
Test dataset: update with new ORC naming scheme
seirl committed rDGRPH1692076e603f: Compression: clean up temporary files, minor fixes (authored by seirl).
Compression: clean up temporary files, minor fixes
May 8 2022
May 8 2022
May 6 2022
May 6 2022
seirl moved T3302: Write docstrings for each method in swh/graph/backend.py from Backlog to Deployed on the Compressed graph service board.
seirl moved T4114: Add logging by default to swh-graph compression from Backlog to Deployed on the Compressed graph service board.
seirl committed rDGRPHbd67a8896bba: Compress: use parallel ScatteredArcsORCGraph instead of converting to ASCII (authored by seirl).
Compress: use parallel ScatteredArcsORCGraph instead of converting to ASCII
seirl requested review of D7760: Compress: use parallel ScatteredArcsORCGraph instead of converting to ASCII.
seirl committed rDGRPH2b3ed4afcec9: generate_dataset.py: only rmtree() output directories if they exist (authored by seirl).
generate_dataset.py: only rmtree() output directories if they exist
seirl closed T4114: Add logging by default to swh-graph compression as Resolved by committing rDGRPH464fded2cd8b: Add logging to compression pipeline.
Add logging to compression pipeline
Thanks for the review!
Address olasd's review
Upgrade dsiutils/sux4j
seirl committed rDGRPHb3128249f7a9: Add checks to prevent crashing on missing node properties (authored by seirl).
Add checks to prevent crashing on missing node properties
seirl committed rDGRPH5ffb3f39f9fd: ExtractNodes: spawn many sort(1) in parallel to avoid locking, then a sort -m… (authored by seirl).
ExtractNodes: spawn many sort(1) in parallel to avoid locking, then a sort -m…
ExtractNodes: read ORC files in parallel
May 4 2022
May 4 2022
May 3 2022
May 3 2022
Fix sort buffer size argument (add b suffix)
Compute sane default for RAM usage
seirl committed rDGRPH2998fc43f82c: Add ScatteredArcsORCGraph, which can read ORC edges in parallel (authored by seirl).
Add ScatteredArcsORCGraph, which can read ORC edges in parallel
seirl committed rDGRPHd5b3b0768d63: Read ORCGraphDataset files in parallel when called from a ForkJoinPool (authored by seirl).
Read ORCGraphDataset files in parallel when called from a ForkJoinPool
seirl committed rDGRPH6626f0a86a73: LabelMapBuilder: parallel read of ORC edges into batches (authored by seirl).
LabelMapBuilder: parallel read of ORC edges into batches
seirl committed rDGRPH654ff3139662: LabelMapBuilder: use a parallelQuickSort to reestablish local order after heap… (authored by seirl).
LabelMapBuilder: use a parallelQuickSort to reestablish local order after heap…
seirl committed rDGRPH597b13e66507: Add -m/--mapped option to graph reading tools (authored by seirl).
Add -m/--mapped option to graph reading tools
May 1 2022
May 1 2022
seirl closed T1848: refresh graph dataset export, a subtask of T3085: Complete and updated copy of the archive on S3 (objects+graph), as Resolved.
Now that there is both a columnar+compressed graph from 2021 and a columnar graph from 2022 that is pending compression, this task about "refreshing the export from January 2019" is resolved.
Apr 29 2022
Apr 29 2022
seirl changed the status of T1848: refresh graph dataset export, a subtask of T3085: Complete and updated copy of the archive on S3 (objects+graph), from Open to Work in Progress.