HomeSoftware Heritage

implement listers as plugins

Description

implement listers as plugins

Listers are declared as plugins via the swh.workers entry_point.

As such, the registry function is expected to return a dict with the
task_modules field (as for generic worker plugins), plus:

  • lister: the lister class,
  • models: list of SQLAlchemy models used by this lister,
  • init (optionnal): hook (callable) used to initialize the lister's state (typically, create/initialize the database for this lister). If not set, the default implementation creates database tables (after optionally having deleted exisintg ones) according to models declared in the models register field.

There is no need for explicitely add lister task modules in the main
conftest module, but any new/extra lister to be tested must be registered
(the tested lister module must be properly installed in the test environment).

Also refactor a bit the cli tools:

  • add support for the standard --config-file option at the 'lister' group level,
  • move the --db-url to the 'lister' group,
  • drop the --lister option for the swh lister db-init cli tool: initializing (especially with --drop-tables) the database for a single lister is unreliable, since all tables are created using a sibgle MetaData (in the same namespace).

Details

Provenance
douarddaAuthored on Sep 3 2019, 3:01 PM
douarddaPushed on Sep 4 2019, 2:41 PM
Differential Revision
D1504: implement listers as plugins
Parents
rDLSc67a926f266a: npm: make NpmVisitModel use the main declarative base class from core.models
Branches
Unknown
Tags
Unknown
Build Status
Buildable 7650
Build 10974: test-and-buildJenkins