Differential D6201 Diff 22668 docs/architecture/metadata.rst

Changeset View

Standalone View

docs/architecture/metadata.rst

This file was added.

				.. _architecture-metadata:

				Metadata workflow and architecture
				==================================

				\|swh\| calls "metadata" information it collects and extracts that describes and provides additional information on source code.

				moraneggUnsubmitted Done Inline Actions intrinsic metadata is part of the code. I think we should focus what metadata is, and not what it is not. \|swh\| calls "metadata" information it collects and extracts that describes and provides additional information on the source code itself. moranegg: intrinsic metadata is part of the code. I think we should focus what metadata is, and not what…
				This metadata is partitioned into three types:

				anlambertUnsubmitted Done Inline Actions Should not it be plural here ? These metadata are partitioned ... anlambert: Should not it be plural here ? These metadata are partitioned ...
				vlorentzAuthorUnsubmitted Done Inline Actions "(meta)data" is uncountable, so it's usually singular. vlorentz: "(meta)data" is uncountable, so it's usually singular.
				1. development metadata, which is part of the :ref:`data-model`, such as authorship
				and date of revisions and releases,
				2. :term:`intrinsic metadata`, which is extracted from a source code repository itself,
				usually mined from metadata files like :file:`package.json` or :file:`Gemfile`.
				moraneggUnsubmitted Done Inline Actions switch collected with extracted. I propose to keep collect for the actions of gathering from external resources. moranegg: switch collected with extracted. I propose to keep collect for the actions of gathering from…
				It is intrinsically part of the software origin, because both are distributed
				together from the origin's VCS repository or release tarballs.
				3. :term:`extrinsic metadata`, which is collected or deposited from external sources.
				It can have a straightforward relationship with the repository (eg. number of stars
				moraneggUnsubmitted Done Inline Actions here I would suggest not to define with the negative statement :term:`extrinsic metadata`, which is collected or deposited from external sources. moranegg: here I would suggest not to define with the negative statement ``` :term:`extrinsic metadata`…
				of GitHub origins or checksums of release tarballs),
				or be more distant (provided by a third-party like Wikidata).

				This document is only about the latter two.


				Raw metadata storage
				--------------------

				As an archive, \|swh\| chooses to store original metadata objects unmodified
				in its long-term storage databases (:ref:`swh-storage <swh-storage>` and
				:ref:`swh-objstorage <swh-objstorage>`).

				For intrinsic metadata, this only means it is treated as any other source code content;
				ie. there is no difference between a metadata file like :file:`package.json`
				and a source code file like :file:`index.js` from the loaders' and the database's
				points of view.

				Extrinsic metadata, however, are stored in a :ref:`dedicated storage service
				<extrinsic-metadata-specification>` (in practice, this is currently in the same database
				as the :ref:`data-model`'s Merkle DAG; but in separate tables).

				As they are both stored verbatim, they are in various formats depending on their source,
				and are not directly usable.


				Indexed metadata storages
				-------------------------

				\|swh\| also stores metadata in indexed databases, which are directly usable
				for searching and querying.
				jayeshvUnsubmitted Not Done Inline Actions An indexer work only on stored/archived metadata? Can't we have an indexer to store number of forks or stars in a github repo? jayeshv: An indexer work only on stored/archived metadata? Can't we have an indexer to store number of…
				vlorentzAuthorUnsubmitted Done Inline Actions No; indexers are not supposed to access the internet. Instead, we'll need to add some sort of loader that fetches and stores API responses; and indexers would read the stored responses vlorentz: No; indexers are not supposed to access the internet. Instead, we'll need to add some sort of…
				Currently, there are two:

				1. the "indexer storage", a postgresql database that acts as a cache, and provides
				limited search functionality
				2. :ref:`swh-search <swh-search>`, an advanced search service backed by
				`ElasticSearch`_.

				Each of these databases has a consistent schema for ease of use.


				Differences between raw and indexed metadata
				--------------------------------------------

				moraneggUnsubmitted Done Inline Actions Add: The raw metadata is the authentic piece of metadata while the indexed metadata is a processed version, where the raw metadata is translated to a uniform vocabulary. Both intrinsic and extrinsic metadata can be indexed and translated. moranegg: Add: ``` The raw metadata is the authentic piece of metadata while the indexed metadata is a…
				The raw metadata is the authentic piece of metadata while the indexed metadata
				is a processed version, where the raw metadata is translated to a uniform vocabulary.

				Both intrinsic and extrinsic metadata can be indexed and translated.

				moraneggUnsubmitted Done Inline Actions Add: By keeping the raw metadata we ensure the possibility to re-compute the metadata in the future with other vocabularies. Furthermore, if we did not store the raw metadata, this would mean bugs in indexers.... (continue with the sentence in text) moranegg: Add: ``` By keeping the raw metadata we ensure the possibility to re-compute the metadata in…
				Therefore, most metadata stored twice in \|swh\|: raw and indexed.
				The reason for this apparent duplication is robustness and future-proofing.

				Indeed, indexing metadata is a complex process.
				By keeping the raw metadata we ensure the possibility to re-compute the metadata
				moraneggUnsubmitted Done Inline Actions I would put this section before the metadata mining as an introduction to it moranegg: I would put this section before the metadata mining as an introduction to it
				in the future with other vocabularies.
				Furthermore, if we did not store the raw metadata, this would mean bugs in indexers
				could easily lose data, forever.
				moraneggUnsubmitted Done Inline Actions delete `as we saw above` moranegg: delete `as we saw above`
				Thanks to this redundant architecture, bugs can be fixed and indexers re-ran
				moraneggUnsubmitted Not Done Inline Actions The raw metadata is the authentic piece of metadata while the indexed metadata is a processed version, where the raw metadata is translated to a uniform vocabulary. both intrinsic and extrinsic metadata can be indexed and translated. moranegg: The raw metadata is the authentic piece of metadata while the indexed metadata is a processed…
				from the raw metadata to fix the indexed metadata.

				moraneggUnsubmitted Done Inline Actions drop `and is not bug free` moranegg: drop `and is not bug free`
				vlorentzAuthorUnsubmitted Done Inline Actions why? vlorentz: why?
				moraneggUnsubmitted Done Inline Actions because of the explanation in the next comment. nothing is bug free :-) You can keep it, but this is not the main reason for keeping raw metadata. moranegg: because of the explanation in the next comment. nothing is bug free :-) You can keep it, but…
				This also makes it easier to add features on metadata mining or change schema
				in the future: instead of re-loading
				moraneggUnsubmitted Done Inline Actions it's not only because of bugs, we keep both because the information is different, we don't translate all properties, etc. also we might choose a different vocabulary in the future, we weren't sure CodeMeta was the best option in the first place. it is a choice that is more about the overall robustness, not just dealing with bugs. moranegg: it's not only because of bugs, we keep both because the information is different, we don't…
				from original sources (which may have disappeared since!), new indexers can simply
				read stored metadata into new indexed storages.


				Metadata mining
				---------------

				Some of the stored raw metadata is read and interpreted by worker processes known
				as :ref:`indexers <swh-indexer>`.
				Currently, they convert this metadata into a common format, `CodeMeta`_.

				Some indexers also read source code files to generate metadata about these files,
				such as their license, language, etc.

				Then, they either send their results directly to a caller, or write it to an
				indexed metadata storage (either directly or through :ref:`swh-journal <swh-journal>`).

				.. _CodeMeta: https://codemeta.github.io/
				.. _ElasticSearch: https://www.elastic.co/elasticsearch/