maybe we should add that metadata on registries is 'extrinsic' metadata

and may discover metadata as a side-effect

this is a bit weird, not sure it is necessary. The list is described as

We define five types of metadata providers:

but today listers are not really metadata providers, unless we consider origins as metadata.

Wouldn't be simpler to not consider a lister as a metadata provider? It looks like it's in fact a 'gatherer', according the definition below, that is described here.

Then (but it is an implementation detail not an architectural aspect), both the gatherer and the lister could be done at the same time by the same worker.

"origins" here has not been defined, nor it has been said that origins are in fact what a lister produces. Adding a sentence in the 'lister' definition stating this fact.

They may either discover metadata as a side-effect of loading source code, or be dedicated to fetching metadata.

Kinda same as above: we are defining these components (i.e. what they are, not what they can be) so this 'it could be used to do other sutff "as a side effect"' should not be in there. This "prospective" part should be in a dedicated paragraph/list/section.

meaning

is a bit weird here. It does not appears like we are defining the deposit as a metadata provider.

you describe the list as

defined by these three properties:

but only list 2 elements.

And the 'name' seems to correspond to the 'type' column in the examples below.

The type/name/instance/xxx family of terms needs to be clarified IMHO.

Do we really want these table in the storage?

this depends on the component (yes implementation detail) is in use when retrieving metadata.
If the component is a lister- lister_gitlab for example is the mechanism with which we fetched the metadata.

If we decide that a new component is needed it will be a new provider_type.

Maybe keeping this potential provider_types to help decide which should be implemented.

For now the only provider_type that is not a "prospective" is deposit_client

It isn't a limited list for now, we have at the moment only one provider_type deposit_client, but the metadata provider entity should be used in the following future cases:

when listing: lister type
when loading: loader type
when fetching metadata from registries: registry type

Because it is not implemented, I'm not sure if we should list the types above in the docs

https://forge.softwareheritage.org/D637#inline-3321

maybe should be:

:term:deposit clients <deposit>, are the clients of the deposit component that push metadata to SWH using credentials; usually at the same time as a :term:software artifact

You are right.

• type and name are mixed up in the table below
  • type is 'deposit_client'
  • name is 'hal'

one = name
two = url
third = type
Where type is used to qualify the metadata, for example, deposit_client metadata is more certified than registry metadata

today listers are not really metadata providers,

Indeed they are not. This spec describes what we want to do, not the current state.

It looks like it's in fact a 'gatherer', according the definition below, that is described here.

"gatherer" is a concept I introduce as a catch-all for everything that is neither a lister, loader, or deposit. I did not yet check how much metadata listers and loaders can fetch. If it happens they can't get a lot of metadata, I'll drop them from the list and replace them with gatherers.

(EDIT: like moranegg just said ^^)

origins are part of the SWH data model, which I assume known by readers of the spec. I should probably mention it at the beginning.

Yeah, I should say "two properties". I used to count the type, but it turns out it's not part of the unique identifier.

Yes. They will be part of the archive. (And they already are.)

Still type and name are mixed up...
deposit_client is type
hal is name

when registry is type and
wikidata is name

switch

switch

all the rest are in the correct order

is information about a software artifact that is not included
in the source code itself but should describe different aspects of the software artifact.
We distinguish attached extrinsic metadata found with the software artifact (on the web view of a repository's forge and/or its API, the metadata attached to a deposit)
and independent extrinsic metadata found in other locations (such as software catalogs and registries).

I changed the text a bit, and introducing two new terms:

• attached extrinsic metadata
• independent extrinsic metadata

With this new remark, if accepted by @zack will affect the rest of the document to divide specification between these 2 categories.

in a format specific to each provider type

I think we should use the same metadata schema for each provider type
This is important when we will document information about specific registries.

I'm reading this paragraph over and over and this is ambiguous.
We don't presume to have th authority to say this is true and this is false.
we say this is a fact:

We found on this day at this location from this provider this metadata

as I written in the new comment, when we distinguish between attached extrinsic metadata and independent extrinsic metadata, we can "qualify" the attached extrinsic metadata as more authoritative.

I would erase this paragraph. Unless there is another way to say that we are not the authority.

add raw

swh-storage should store data as close to the original source as possible.
Unification/translation should be done as an indexer, so it can be re-started based on the content of swh-storage.

If we did the translation before sending to swh-storage, then a bug in the translation means all existing data is corrupted.

Where?

I now understand why you thought I didn't want the raw metadata, but here we discuss the information about the provider and not the fetched metadata.
This is metadata about the provider.