Change Details

Using a linux commit (`swh:1:rev:f39d7d78b70e0f39facb1e4fab77ad3df5c52a35`), doing a swh-graph API call `graph/visit/nodes/{SWHID}?edges=rev:rev` takes around 40s to complete/process all data. We should benchmark/investigate different solutions, such as: - Creating the symlinks incrementally as soon as you receive content from swh-graph API - Sharding

Using a linux commit (`swh:1:rev:f39d7d78b70e0f39facb1e4fab77ad3df5c52a35`), doing a swh-graph API call `graph/visit/nodes/{SWHID}?edges=rev:rev` takes around 40s to complete/process all data, because we wait for all the history to be retrieved before starting to process it. We should instead process it in a streaming fashion, starting to populate history/ incrementally, and using .status file to inform the user about progress.

Using a linux commit (`swh:1:rev:f39d7d78b70e0f39facb1e4fab77ad3df5c52a35`), doing a swh-graph API call `graph/visit/nodes/{SWHID}?edges=rev:rev` takes around 40s to complete/process all data, because we wait for all the history to be retrieved before starting to process it. We should benchmark/investigate different solutionsinstead process it in a streaming fashion, such as: - Creating the symlinksstarting to populate history/ incrementally as soon as you receive content from swh-graph API - Sharding, and using .status file to inform the user about progress.