Maniphest T4080

Minimize archival lag w.r.t. upstream code hosting platforms
Open, HighPublic
Actions

Assigned To

None

Authored By

	bchauvet
	Mar 23 2022, 4:34 PM

Tags

Subscribers

Description

Quantify and monitor in real-time the lag, especially for major platforms (GitHub, GitLab.com, etc.) - T4130
Improve ingestion efficiency (optimize loaders, especially the Git loader, optimize scheduling policies) - T2207
Make lag monitoring dashboards easy to find (for decision makers) - T4133

Related Objects
Search...

		Status	Assigned	Task
		Open	None	T4080 Minimize archival lag w.r.t. upstream code hosting platforms
		Open	None	T4130 Quantify and monitor in real-time the lag, especially for major platforms
		Open	None	T2207 Improve ingestion efficiency
		Work in Progress	olasd	T846 Some objects from the original GitHub import have never actually been imported.
		Open	None	T3655 loader git: enable global deduplication of head branches before fetching them
		Open	None	T3635 git loader: enable "partial" global deduplication of revisions via the extid mapping table
		Open	None	T3654 loader git: load revisions in topological order
		Open	None	T3656 Survey revisions/releases with partially loaded history
		Open	None	T3660 Nodes with missing ancestors in SWH DAG / SWH-graph
		Resolved	olasd	T3544 Deal with GitHub removing support for git:// URLs
		Open	None	T4133 Make lag monitoring dashboards easy to find
		Open	None	T4185 Loader profiling : Add Measure of ignored objects

Event Timeline

bchauvet created this task.Mar 23 2022, 4:34 PM

bchauvet triaged this task as High priority.Mar 25 2022, 5:22 PM

grafana scheduler metrics dashboard may help [1]

[1] https://grafana.softwareheritage.org/goto/XHduArynk?orgId=1

bchauvet updated the task description. (Show Details)Apr 11 2022, 11:44 AM

bchauvet added a subtask: T2207: Improve ingestion efficiency .

bchauvet updated the task description. (Show Details)Apr 11 2022, 11:48 AM

grooming meeting on 2022/04/06 : https://hedgedoc.softwareheritage.org/WQaWrtUSTo6w-v7US6yRUw# (notes added in the corresponding tasks)