Page MenuHomeSoftware Heritage

listers: Add proper integration tests to lister
Closed, MigratedEdits Locked

Description

Currently the lister are partially tested, mostly through mocks (and not easily debugged when something goes wrong).

Sometimes, after a refactoring, they do not break because the coverage is not enough.

Now that a swh-listers fixture got introduced (D2077), we should at least add one integration test per lister for the basic listing use case.

Plan:

  • all listers: 1 basic listing (checking the lister's output in the scheduling db)

Ideally, adding those would be best:

  • all listers: 1 basic listing failure (e.g lister misconfiguration ; phabricator lister requires api.token set or otherwise it fails, etc...)
  • all listers: 1 listing with filtering on already seen origins
  • pagination listers: multiple page calls (npm does now)

Related Objects

Mentioned In
rDLSa64ae9641d53: debian.lister: Add integration test which checks scheduled tasks
rDLSb4867ccda965: npm.tests: Add an integration test on listing with pagination
D2139: npm.tests: Add an integration test on listing with pagination
D2134: debian.lister: Add integration test which checks scheduled tasks
D2133: pytest_plugin: Add support for http request (debian lister related)
rDLS1889875f67cc: gitlab.lister: Add integration test which checks scheduled tasks
rDLS0b8b1419e14a: github.lister: Add integration test which checks scheduled tasks
rDLSf92ac8364612: bitbucket.lister: Add integration test which checks scheduled tasks
rDLS903b644c6348: phabricator.lister: Add integration test which checks scheduled tasks
rDLSf3bf9ae50f7d: packagist.lister: Add integration test which checks scheduled tasks
D2128: phabricator.lister: Add integration test which checks scheduled tasks
D2127: packagist.lister: Add integration test which checks scheduled tasks
rDLS678b7ea5bd51: npm.lister: Add integration test which checks the scheduled tasks
rDLS599af25ad6d6: pypi.lister: Add integration test which checks the scheduled tasks
rDLS8d50e0d9410f: cran.lister: Fix cran lister and add proper integration test
D2122: npm.lister: Add integration test which checks the scheduled tasks
D2121: pypi.lister: Add integration test which checks the scheduled tasks
D2120: cran.lister: Fix cran lister and add proper integration test
Mentioned Here
D2077: cgit.tests: Check the tasks from the scheduler
D2074: gnu.tests: Check gnu lister
D2076: gnu: Separate listing and parsing logic and add integration test

Event Timeline

ardumont triaged this task as Normal priority.Oct 8 2019, 9:06 AM
ardumont created this task.

Related D2074 D2076 D2077

ardumont changed the task status from Open to Work in Progress.Oct 13 2019, 6:37 PM
ardumont claimed this task.