Page MenuHomeSoftware Heritage

Publish status of existing listers and loaders
Closed, MigratedEdits Locked

Description

This is a meta-task to track activities related to exposing the status of existing listers and loaders.
The goal is to, on one side, make public our progress, and on the other side, to make it easier to external contributors to see what has already been done and what is missing.

This involves the following:

  • design web page
  • process to maintain up to date
  • make this information clearly visible and link to Sloan subgrants

Related Objects

StatusAssignedTask
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration
Migratedgitlab-migration

Event Timeline

vlorentz triaged this task as Normal priority.Mar 15 2021, 12:28 PM
bchauvet raised the priority of this task from Normal to High.Mar 25 2022, 5:26 PM

As that documentation of loaders and listers currently available for the archive will target a general audience and not a technical one,
it will be added in the Software Heritage User documentation.

Two new sections Software Heritage loaders and Software Heritage listers will be added.
One subsection per loader/lister will document it in a high-level overview way.
At the top of each new section, a grid of logos related to the implemented loaders (VCS, source packages, ...)
and listers (forges, package managers, ...) will be displayed linking to the related subsections.

For a loader, the following content should be found in its documentation:

  • description of the source of software artifacts to load into the archive (VCS tool introduction for instance)
  • link to source code repository and mention of related grants and external contributors if any
  • link to current related issues on our forge
  • high-level overview of how the loader process data to load into the archive
  • how the found software artifacts are mapped to the SWH data model
  • noticeable implemented features (for instance dump protocol support in git loader or externals processing in svn loader)
  • reference mapping of external ids to SWHIDs if any
  • description of extracted intrinsic metadata if any
  • description of loader parameters
  • example of invocation in the docker environment for quick user testing
  • set of listers producing tasks that will execute the loader
  • current deployment state (in production ? in staging ?)

For a lister, the following content should be found in its documentation:

  • description of the source of software origins listing (forge, package manager, ...)
  • link to source code repository and mention of related grants and external contributors if any
  • link to current related issues on our forge
  • high-level overview of the process to discover origins
  • description of last update date for an origin if any
  • description of lister parameters
  • description of lister credentials (required ? recommended ?)
  • example of invocation in the docker environment for quick user testing
  • set of loaders that will be used to ingest listed origins into the archive
  • current deployment state (in production ? in staging ?)
anlambert changed the task status from Open to Work in Progress.May 10 2022, 4:29 PM
gitlab-migration closed subtask T4153: Document cvs loader as Migrated.
gitlab-migration closed subtask T4156: Document cran loader as Migrated.
gitlab-migration closed subtask T4161: Document npm loader as Migrated.
gitlab-migration closed subtask T4162: Document opam loader as Migrated.
gitlab-migration closed subtask T4163: Document pypi loader as Migrated.
gitlab-migration closed subtask T4165: Document cgit lister as Migrated.
gitlab-migration closed subtask T4166: Document cran lister as Migrated.
gitlab-migration closed subtask T4172: Document gnu lister as Migrated.
gitlab-migration closed subtask T4175: Document npm lister as Migrated.
gitlab-migration closed subtask T4176: Document opam lister as Migrated.
gitlab-migration closed subtask T4179: Document pypi lister as Migrated.