Page MenuHomeSoftware Heritage

history/ dir browsing is too slow on big repos like the Linux kernel
Open, NormalPublic

Description

Using a linux commit (swh:1:rev:f39d7d78b70e0f39facb1e4fab77ad3df5c52a35), doing a swh-graph API call graph/visit/nodes/{SWHID}?edges=rev:rev takes around 40s to complete/process all data.

We should benchmark/investigate different solutions, such as:

  • Creating the symlinks incrementally as soon as you receive content from swh-graph API
  • Sharding

Event Timeline

haltode triaged this task as Normal priority.Oct 20 2020, 4:04 PM
haltode created this task.
haltode created this object in space S1 Public.
zack renamed this task from Realistic benchmark to explore the Linux kernel history to history/ dir browsing is too slow on big repos like the Linux kernel.Oct 20 2020, 6:23 PM