Maniphest T4080

Minimize archival lag w.r.t. upstream code hosting platforms
Closed, MigratedEdits Locked
Actions

Assigned To

gitlab-migration

Authored By

	bchauvet
	Mar 23 2022, 4:34 PM

Tags

Subscribers

gitlab-migration

Description

Quantify and monitor in real-time the lag, especially for major platforms (GitHub, GitLab.com, etc.) - T4130
Improve ingestion efficiency (optimize loaders, especially the Git loader, optimize scheduling policies) - T2207
Make lag monitoring dashboards easy to find (for decision makers) - T4133

Related Objects
Search...

		Status	Assigned	Task
		Migrated	gitlab-migration	T4080 Minimize archival lag w.r.t. upstream code hosting platforms
		Migrated	gitlab-migration	T4130 Quantify and monitor in real-time the lag, especially for major platforms
		Migrated	gitlab-migration	T2207 Improve ingestion efficiency
		Migrated	gitlab-migration	T846 Some objects from the original GitHub import have never actually been imported.
		Migrated	gitlab-migration	T3655 loader git: enable global deduplication of head branches before fetching them
		Migrated	gitlab-migration	T3635 git loader: enable "partial" global deduplication of revisions via the extid mapping table
		Migrated	gitlab-migration	T3654 loader git: load revisions in topological order
		Migrated	gitlab-migration	T3656 Survey revisions/releases with partially loaded history
		Migrated	gitlab-migration	T3660 Nodes with missing ancestors in SWH DAG / SWH-graph
		Migrated	gitlab-migration	T3544 Deal with GitHub removing support for git:// URLs
		Migrated	gitlab-migration	T4133 Make lag monitoring dashboards easy to find
		Migrated	gitlab-migration	T4185 Loader profiling : Add Measure of ignored objects

Event Timeline

bchauvet created this task.Mar 23 2022, 4:34 PM

bchauvet triaged this task as High priority.Mar 25 2022, 5:22 PM

grafana scheduler metrics dashboard may help [1]

[1] https://grafana.softwareheritage.org/goto/XHduArynk?orgId=1

bchauvet updated the task description. (Show Details)Apr 11 2022, 11:44 AM

bchauvet added a subtask: T2207: Improve ingestion efficiency .

bchauvet updated the task description. (Show Details)Apr 11 2022, 11:48 AM

grooming meeting on 2022/04/06 : https://hedgedoc.softwareheritage.org/WQaWrtUSTo6w-v7US6yRUw# (notes added in the corresponding tasks)

gitlab-migration closed subtask T4185: Loader profiling : Add Measure of ignored objects as Migrated.Jan 8 2023, 5:04 PM

gitlab-migration closed subtask T2207: Improve ingestion efficiency as Migrated.Jan 8 2023, 10:22 PM

This task has been migrated to GitLab.

gitlab-migration closed subtask T4130: Quantify and monitor in real-time the lag, especially for major platforms as Migrated.Jan 8 2023, 10:24 PM

gitlab-migration closed subtask T4133: Make lag monitoring dashboards easy to find as Migrated.