HomeSoftware Heritage

journal_client: Compute next position for origin visit

Description

journal_client: Compute next position for origin visit

For origin without any last_update information [1], the journal client is now also in
charge of moving their next position in the queue for rescheduling. Depending on their
status, the next position offset and next_visit_queue_position are updated after each
visit completes:

  • if the visit has failed, increase the next visit target by the minimal visit interval (to take into account transient loading issues)
  • if the visit is successful, and records some changes, decrease the visit interval index by 2 (visit the origin *way* more often).
  • if the visit is successful, and records no changes, increase the visit interval index by 1 (visit the origin less often).

We then set the next visit target to its current value + the new visit interval
multiplied by a random fudge factor (picked in the -/+ 10% range).

The fudge factor allows the visits to spread out, avoiding "bursts" of loaded origins
e.g. when a number of origins from a single hoster are processed at once.

Note that the computations happen for all origins for simplicity and code maintenance
but it will only be used by a new soon-to-be scheduling policy.

[1] Lister cannot provide it for some reason.