HomeSoftware Heritage

simulator: stop using the database as a cache for origin data

Description

simulator: stop using the database as a cache for origin data

This was a significant bottleneck of the simulator. To work around this,
we:

  • Generate snapshot ids consistently in the OriginModel
  • Cache the origin data locally in the simulator, to compute the eventfulness of visits
  • Cache the last visit time for all origins to compute the estimated run time of visit tasks.

Details

Provenance
olasdAuthored on Jan 21 2021, 5:45 PM
vlorentzCommitted on Jan 26 2021, 1:31 PM
vlorentzPushed on Jan 29 2021, 10:00 AM
Differential Revision
D4914: simulator: stop using the database as a cache for origin data
Parents
rDSCHc92ead5875ec: grab_next_visits: don't re-schedule visits too fast
Branches
Unknown
Tags
Unknown
Build Status
Buildable 18879
Build 29246: test-and-buildJenkins console · Jenkins