HomeSoftware Heritage

Make the SourceForge lister incremental

This commit no longer exists in the repository. It may have been part of a branch which was deleted.

Description

Make the SourceForge lister incremental

SourceForge's sitemaps (1 main one + many sharded) give us a "last
modified" date for every subsitemap and project, allowing us to perform
an incremental listing.

We store the subsitemaps' "last modified" dates in the lister state, as
well as those of the empty projects (projects which don't have any VCS
registered), and the rest comes from the already visited origins from
the database.

The tests try to cover the possible cases of a subsitemap that has
changed, one that hasn't, a project that has change, one that hasn't,
and same for an empty project.

Details

Provenance
AlphareAuthored on Apr 30 2021, 9:46 PM
AlpharePushed on May 6 2021, 10:32 AM
Differential Revision
D5659: Make the SourceForge lister incremental
Build Status
Buildable 21310
Build 33095: test-and-buildJenkins console · Jenkins

Commit No Longer Exists

This commit no longer exists in the repository.