Page MenuHomeSoftware Heritage

Package up the common logic for the GitHub lister in a common class
Closed, ResolvedPublic

Description

The GitHub lister will serve as base for other listers. Some operations need to be factored out and cleaned up for other listers to be able to reuse/inspire themselves of.

  • Database schema management
    • à la swh.storage: one full SQL file with schema version + incremental updates
  • Initialization
    • Connection to the main database (origins)
    • Connection to the scheduler database (tasks)
  • Origin/Task creation (incremental and full operations)
    • Mass create tasks and get ids
    • Mass create origins and get ids
  • Task disabling (full operations)
    • Mass disable tasks from ids

We obviously need to make sure those operations exist in the underlying libraries (swh.storage, swh.scheduler)

Event Timeline

The common logic has been packaged up in swh.storage and swh.scheduler.

The GitHub lister now inherits from a base class, swh.lister.github.base.SWHLister, that should move to a common module once we develop a new lister.