Page MenuHomeSoftware Heritage
Create Task
Maniphest T1847

fully automate export of the graph dataset
Closed, MigratedEdits Locked
Actions

Description

we want to fully automate the export of the archive graph, see most recent output example here: https://annex.softwareheritage.org/public/dataset/graph/2019-01-28/

Related Objects
Search...

Event Timeline

zack triaged this task as High priority.Jun 23 2019, 10:20 PM
zack created this task.
zack mentioned this in T1848: refresh graph dataset export.
zack added a parent task: T1848: refresh graph dataset export.
zack lowered the priority of this task from High to Normal.Nov 18 2019, 2:48 PM
zack added a project: Compressed graph service.
zack added a comment.Sep 16 2020, 8:43 PM

I think this is (reasonably) done now, please check and close it.

seirl added a comment.Sep 16 2020, 10:59 PM

No, only the edge part is done, we still need a parquet and a CSV exporter :/

zack changed the task status from Open to Work in Progress.Sep 17 2020, 9:04 AM
seirl closed this task as Resolved.Mar 26 2021, 12:04 PM

The ORC exporter is done, and it's likely that we won't provide CSV exports in the future, or we'll generate them from the ORC format.

zack reopened this task as Open.Mar 26 2021, 12:25 PM

reopening, as ideally we'd like to have run the entire ORC export once to completion before closing

seirl added a subtask: T2431: Document how to export the graph edge dataset.Apr 7 2021, 3:03 PM
seirl closed subtask T3178: document how to export the graph dataset automatically as Invalid.
seirl closed this task as Resolved.Apr 29 2022, 5:57 PM

Done!

seirl closed subtask T2431: Document how to export the graph edge dataset as Resolved.Apr 29 2022, 6:15 PM
seirl moved this task from Backlog to Deployed on the Compressed graph service board.Apr 29 2022, 6:22 PM
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 9:59 PM
gitlab-migration claimed this task.
gitlab-migration added subscribers: seirl, gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T2431: Document how to export the graph edge dataset from Resolved to Migrated.Jan 8 2023, 10:00 PM
gitlab-migration changed the status of subtask T3178: document how to export the graph dataset automatically from Invalid to Migrated.
About · Developer information · Legal