HomeSoftware Heritage

implement listers as plugins

This commit no longer exists in the repository. It may have been part of a branch which was deleted.

Description

implement listers as plugins

Listers are declared as plugins via the swh.workers entry_point.

As such, the registry function is expected to return a dict with the
task_modules field (as for generic worker plugins), plus:

  • lister: the lister class,
  • models: list of SQLAlchemy models used by this lister,
  • init (optionnal): hook (callable) used to initialize the lister's state (typically, create/initialize the database for this lister). If not set, the default implementation creates database tables (after optionally having deleted exisintg ones) according to models declared in the models register field.

There is no need for explicitely add lister task modules in the main
conftest module, but any new/extra lister to be tested must be registered
(the tested lister module must be properly installed in the test environment).

Also refactor a bit the cli tools:

  • add support for the standard --config-file option at the 'lister' group level,
  • move the --db-url to the 'lister' group,
  • drop the --lister option for the swh lister db-init cli tool: initializing (especially with --drop-tables) the database for a single lister is unreliable, since all tables are created using a sibgle MetaData (in the same namespace).

Details

Provenance
douarddaAuthored on Sep 3 2019, 3:01 PM
douarddaPushed on Sep 4 2019, 2:41 PM
Differential Revision
D1504: implement listers as plugins
Build Status
Buildable 7650
Build 10974: test-and-buildJenkins

Commit No Longer Exists

This commit no longer exists in the repository.