HomeSoftware Heritage

origin_head: Do not fetch complete snapshots for non-FTP visits

Description

origin_head: Do not fetch complete snapshots for non-FTP visits

Some snapshots are really large. Rather than fetching them entirely only to
discard most of the branches, this commit only fetches some branches (to
check existence + to use less queries on small snapshots), then requests
specific branches as needed (usually only 2).

This should improve performance and reduce timeout exceptions from the
storage.

Details

Provenance
vlorentzAuthored on Nov 21 2022, 1:38 PM
vlorentzPushed on Nov 23 2022, 8:56 AM
Differential Revision
D8861: origin_head: Do not fetch complete snapshots for non-FTP visits
Parents
rDCIDXb7f04dd9d40d: journal writer: only flush kafka once per batch
Branches
Unknown
Tags
Unknown
Build Status
Buildable 32920
Build 51600: test-and-buildJenkins console · Jenkins