HomeSoftware Heritage

Implement storage of listed origins

Authored by olasd on Jun 16 2020, 10:25 AM.

Description

Implement storage of listed origins

This new API endpoint allows listers to record the origins they have seen during
their current run.

Origins are identified by the lister instance, the url of the origin, and the
type of loader that should be used to load this origin.

The implementation allows listers just send the list of origins they've
seen (with some lightweight extra information), leaving the backend to handle
whether to do an insertion or an update to an existing origin.

The current implementation doesn't disable origins that have disappeared when
doing a full listing run. This step will be done by a separate "origin garbage
collection" endpoint, which will peruse the last_seen field.