HomeSoftware Heritage

Make the SourceForge lister incremental

Description

Make the SourceForge lister incremental

SourceForge's sitemaps (1 main one + many sharded) give us a "last
modified" date for every subsitemap and project, allowing us to perform
an incremental listing.

We store the subsitemaps' "last modified" dates in the lister state, as
well as those of the empty projects (projects which don't have any VCS
registered), and the rest comes from the already visited origins from
the database.

The tests try to cover the possible cases of a subsitemap that has
changed, one that hasn't, a project that has change, one that hasn't,
and same for an empty project.

Details

Provenance
AlphareAuthored on Apr 30 2021, 9:46 PM
AlpharePushed on May 6 2021, 10:32 AM
Differential Revision
D5659: Make the SourceForge lister incremental
Parents
rDLS6f8dd5d3f243: tox: Add sphinx environments to check sane doc build
Branches
Unknown
Tags
Unknown
References
tag: v1.2.0
Build Status
Buildable 21310
Build 33095: test-and-buildJenkins console · Jenkins