Page MenuHomeSoftware Heritage

listers: Add proper integration tests to lister
Started, Work in Progress, NormalPublic

Description

Currently the lister are partially tested, mostly through mocks (and not easily debugged when something goes wrong).

Sometimes, after a refactoring, they do not break because the coverage is not enough.

Now that a swh-listers fixture got introduced (D2077), we should at least add one integration test per lister for the basic listing use case.

Plan:

  • all listers: 1 basic listing (checking the lister's output in the scheduling db)

Ideally, adding those would be best:

  • all listers: 1 basic listing failure (e.g lister misconfiguration ; phabricator lister requires api.token set or otherwise it fails, etc...)
  • all listers: 1 listing with filtering on already seen origins
  • pagination listers: multiple page calls (npm does now)

Related Objects

Mentioned In
rDLSa64ae9641d53: debian.lister: Add integration test which checks scheduled tasks
rDLSb4867ccda965: npm.tests: Add an integration test on listing with pagination
D2139: npm.tests: Add an integration test on listing with pagination
D2134: debian.lister: Add integration test which checks scheduled tasks
D2133: pytest_plugin: Add support for http request (debian lister related)
rDLS1889875f67cc: gitlab.lister: Add integration test which checks scheduled tasks
rDLS0b8b1419e14a: github.lister: Add integration test which checks scheduled tasks
rDLSf92ac8364612: bitbucket.lister: Add integration test which checks scheduled tasks
rDLS903b644c6348: phabricator.lister: Add integration test which checks scheduled tasks
rDLSf3bf9ae50f7d: packagist.lister: Add integration test which checks scheduled tasks
D2128: phabricator.lister: Add integration test which checks scheduled tasks
D2127: packagist.lister: Add integration test which checks scheduled tasks
rDLS678b7ea5bd51: npm.lister: Add integration test which checks the scheduled tasks
rDLS599af25ad6d6: pypi.lister: Add integration test which checks the scheduled tasks
rDLS8d50e0d9410f: cran.lister: Fix cran lister and add proper integration test
D2122: npm.lister: Add integration test which checks the scheduled tasks
D2121: pypi.lister: Add integration test which checks the scheduled tasks
D2120: cran.lister: Fix cran lister and add proper integration test
Mentioned Here
D2077: cgit.tests: Check the tasks from the scheduler
D2074: gnu.tests: Check gnu lister
D2076: gnu: Separate listing and parsing logic and add integration test

Event Timeline

ardumont triaged this task as Normal priority.Tue, Oct 8, 9:06 AM
ardumont created this task.

Related D2074 D2076 D2077

ardumont updated the task description. (Show Details)Thu, Oct 10, 9:50 AM
ardumont updated the task description. (Show Details)Sat, Oct 12, 2:59 AM
ardumont changed the task status from Open to Work in Progress.Sun, Oct 13, 6:37 PM
ardumont updated the task description. (Show Details)Fri, Oct 18, 9:55 AM