Page MenuHomeSoftware Heritage

Publish status of existing listers and loaders
Started, Work in Progress, HighPublic

Description

This is a meta-task to track activities related to exposing the status of existing listers and loaders.
The goal is to, on one side, make public our progress, and on the other side, to make it easier to external contributors to see what has already been done and what is missing.

This involves the following:

  • design web page
  • process to maintain up to date
  • make this information clearly visible and link to Sloan subgrants

Event Timeline

vlorentz triaged this task as Normal priority.Mar 15 2021, 12:28 PM
bchauvet raised the priority of this task from Normal to High.Mar 25 2022, 5:26 PM

As that documentation of loaders and listers currently available for the archive will target a general audience and not a technical one,
it will be added in the Software Heritage User documentation.

Two new sections Software Heritage loaders and Software Heritage listers will be added.
One subsection per loader/lister will document it in a high-level overview way.
At the top of each new section, a grid of logos related to the implemented loaders (VCS, source packages, ...)
and listers (forges, package managers, ...) will be displayed linking to the related subsections.

For a loader, the following content should be found in its documentation:

  • description of the source of software artifacts to load into the archive (VCS tool introduction for instance)
  • link to source code repository and mention of related grants and external contributors if any
  • link to current related issues on our forge
  • high-level overview of how the loader process data to load into the archive
  • how the found software artifacts are mapped to the SWH data model
  • noticeable implemented features (for instance dump protocol support in git loader or externals processing in svn loader)
  • reference mapping of external ids to SWHIDs if any
  • description of extracted intrinsic metadata if any
  • description of loader parameters
  • example of invocation in the docker environment for quick user testing
  • set of listers producing tasks that will execute the loader
  • current deployment state (in production ? in staging ?)

For a lister, the following content should be found in its documentation:

  • description of the source of software origins listing (forge, package manager, ...)
  • link to source code repository and mention of related grants and external contributors if any
  • link to current related issues on our forge
  • high-level overview of the process to discover origins
  • description of last update date for an origin if any
  • description of lister parameters
  • description of lister credentials (required ? recommended ?)
  • example of invocation in the docker environment for quick user testing
  • set of loaders that will be used to ingest listed origins into the archive
  • current deployment state (in production ? in staging ?)
anlambert changed the task status from Open to Work in Progress.Tue, May 10, 4:29 PM