swh-graph in production
Open, HighPublic
Actions

Description

We need to have an in production swh-graph service, including fully automated periodic exports from storage.
This is a meta-task to track all the related activities to achieve this goal.

Revisions and Commits

rDGRPH Compressed graph representation
	D8919	rDGRPHb76801259953 Add CLI script to generate Luigi config and call it
	D8891	rDGRPH2b634bf4192d luigi: Simplify compression task
	D8891	rDGRPH0a4b706400d4 luigi: Remove option to configure graph name
	D8891	rDGRPH60e4707feca6 Add luigi task to compress the graph
	D8881	rDGRPHdc2cb79bc670 Document the need for higher vm.max_map_count
	D8880	rDGRPH486312e00eaa Fix crash on null Release message

Related Objects
Search...

Status	Assigned	Task
Open	None	T2201 Indexing / mining
Open	anlambert	T2204 Full-text search on source code (prototype)
Open	None	T2217 Plumbings
Work in Progress	vlorentz	T3096 Efficient and reliable download via the Vault
Open	None	T3550 Compute and show ETA for vault tasks
Work in Progress	None	T887 Vault: "snapshot" cooker
Open	vlorentz	T2220 swh-graph in production
Resolved	seirl	T3161 graph service: add anti-DoS limit on the number of edges traversed
Open	vlorentz	T4676 Add Luigi workflow in swh-dataset

Event Timeline

douardda created this task.Jan 20 2020, 2:28 PM

vlorentz triaged this task as Normal priority.Jan 22 2020, 4:19 PM

vlorentz added a project: Compressed graph service.Jan 22 2020, 4:36 PM

rdicosmo added a project: Roadmap 2021.Mar 8 2021, 9:51 AM

rdicosmo removed a project: Roadmap 2020.Mar 8 2021, 9:57 AM

rdicosmo added a parent task: T3096: Efficient and reliable download via the Vault.Mar 8 2021, 10:08 AM

rdicosmo updated the task description. (Show Details)Mar 8 2021, 10:12 AM

rdicosmo added a project: meta-task.

rdicosmo added a parent task: T2204: Full-text search on source code (prototype).Mar 8 2021, 10:47 AM

rdicosmo moved this task from Backlog to Work in progress on the Roadmap 2021 board.Mar 19 2021, 12:45 PM

zack added a subtask: T3161: graph service: add anti-DoS limit on the number of edges traversed.Mar 22 2021, 9:43 AM

vlorentz assigned this task to zack.Apr 23 2021, 4:50 PM

zack mentioned this in T3639: prepare quote for "granet2", next gen swh-graph compression server.Oct 8 2021, 4:04 PM

vlorentz added a parent task: T3550: Compute and show ETA for vault tasks.Nov 15 2021, 9:55 AM

vlorentz mentioned this in T3550: Compute and show ETA for vault tasks.

zack changed the status of subtask T3161: graph service: add anti-DoS limit on the number of edges traversed from Open to Work in Progress.Nov 23 2021, 1:38 PM

seirl closed subtask T3161: graph service: add anti-DoS limit on the number of edges traversed as Resolved.Jan 12 2022, 5:01 PM

vlorentz added a parent task: T887: Vault: "snapshot" cooker.Feb 8 2022, 2:05 PM

vlorentz mentioned this in T887: Vault: "snapshot" cooker.

bchauvet added a project: Roadmap 2022.Mar 23 2022, 5:07 PM

bchauvet raised the priority of this task from Normal to High.Mar 25 2022, 5:29 PM

Graph status meeting

Interrogation du graph
- Rocquencourt : Granet 700GO de ram (max atteint)

Compression du graph : 1.7TO minimum
- Telecom : machine 4TO

Compression

Dataset: https://docs.softwareheritage.org/devel/swh-dataset/graph/dataset.html?highlight=dataset
pipeline de compression : python
implementation des étapes : org.softwareheritage.grap.compress

TODO

Finish GRPC migration (seirl)
- Forge issues to cleanup once GRPC is merged (seirl)
Automate deployment (sysadm) >> prepare the command
- T3168
Native hadoop libraries (?)
- T4250
- Benchmark perfs with and without the hadoop librairies
Luigi ETL[1] pipeline for compression / deployment
- T3832
Integration of the generated javadoc in swh docs (vlorentz)
- T1971
Integration of Java code coverage in the forge
Unit test the compression pipeline