Differential D3939 Diff 13877 docs/specs/spec-loading.rst

Changeset View

Standalone View

docs/specs/spec-loading.rst

Show First 20 Lines • Show All 46 Lines • ▼ Show 20 Lines
\| directory \| root directory of the expanded submitted\|		\| directory \| root directory of the expanded submitted\|
\| \| tarball \|		\| \| tarball \|
+------------------------------------+-----------------------------------------+		+------------------------------------+-----------------------------------------+


Origin artifact		Origin artifact
~~~~~~~~~~~~~~~		~~~~~~~~~~~~~~~

We create an origin using the url in the deposited metadata.		We create an origin URL by concatenating the client URI and the value of the
The current deposit and future deposits with the same url or external_id		Slug header of the initial POST request of the deposit.
		moraneggUnsubmitted Not Done Inline Actions maybe we should add an example from T2391? not sure it is the right place to do so.. moranegg: maybe we should add an example from T2391? not sure it is the right place to do so..
		vlorentzAuthorUnsubmitted Done Inline Actions We can add a link to it once the documentation exists vlorentz: We can add a link to it once the documentation exists
will be associated to this origin.

.. code-block:: json		.. code-block:: json

{		{
"origin": {		"origin": {
"id": 89283768,		"id": 89283768,
"origin_visits_url": "/api/1/origin/89283768/visits/",		"origin_visits_url": "/api/1/origin/89283768/visits/",
"type": "deposit",		"type": "deposit",
Show All 32 Lines	.. code-block:: json
"visit": 1		"visit": 1
}		}
]		]
}		}

Snapshot artifact		Snapshot artifact
~~~~~~~~~~~~~~~~~		~~~~~~~~~~~~~~~~~

The snapshot represents one deposit push. The master branch points to a		The snapshot represents one deposit push. The ``HEAD`` branch points to a
synthetic revision. We will create a second branch pointing to a release		synthetic revision.
artifact, if the indicate that the deposit is a release with a `releaseNotes`.

.. code-block:: json		.. code-block:: json

{		{
"snapshot": {		"snapshot": {
"branches": {		"branches": {
"master": {		"HEAD": {
"target": "396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52",		"target": "396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52",
"target_type": "revision",		"target_type": "revision",
"target_url": "/api/1/revision/396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52/"		"target_url": "/api/1/revision/396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52/"
}		}
"refs/tags/v1.1": {
"target": "a9f3396f372ed4a51d75e15ca16c1c2df1fc5c97",
"target_type": "release",
"target_url": "/api/1/release/a9f3396f372ed4a51d75e15ca16c1c2df1fc5c97/"
}
},		},
"id": "a3773941561cc557853898773a19c07cfe2efc5a",		"id": "a3773941561cc557853898773a19c07cfe2efc5a",
"next_branch": null		"next_branch": null
}		}
}		}

Release artifact		Note that previous versions of the deposit-loader named the branch ``master``
~~~~~~~~~~~~~~~~		instead, and created release branches under certain conditions.
		moraneggUnsubmitted Done Inline Actions We didn't create, it was in spec but never implemented. We still need to implement this, which is described in T1755. I prefer keeping the `release` in spec and saying it is on the roadmap. moranegg: We didn't create, it was in spec but never implemented. We still need to implement this, which…
		vlorentzAuthorUnsubmitted Done Inline Actions good point vlorentz: good point
The content is deposited with a set of descriptive metadata in the CodeMeta
vocabulary. The following CodeMeta terms implies that the
artifact is a release:

- `releaseNotes`
- `softwareVersion`

If present, a release artifact will be created with the mapping below:

+-------------------+-----------------------------------+-----------------+----------------+
\| SWH release field \| Description \| CodeMeta term \| Fallback value \|
+===================+===================================+=================+================+
\| target \| revision containing all metadata \| X \|X \|
+-------------------+-----------------------------------+-----------------+----------------+
\| target_type \| revision \| X \|X \|
+-------------------+-----------------------------------+-----------------+----------------+
\| name \| release or tag name (mandatory) \| softwareVersion \| X \|
+-------------------+-----------------------------------+-----------------+----------------+
\| message \| message associated with release \| releaseNotes \| X \|
+-------------------+-----------------------------------+-----------------+----------------+
\| date \| release date = publication date \| datePublished \| deposit_date \|
+-------------------+-----------------------------------+-----------------+----------------+
\| author \| deposit client \| author \| client \|
+-------------------+-----------------------------------+-----------------+----------------+


.. code-block:: json

{
"release": {
"author": {
"email": "hal@ccsd.cnrs.fr",
"fullname": "HAL <phal@ccsd.cnrs.fr>",
"name": "HAL"
},
"author_url": "/api/1/person/x/",
"date": "2019-05-27T16:28:33+02:00",
"id": "a9f3396f372ed4a51d75e15ca16c1c2df1fc5c97",
"message": "AffectationRO Version 1.1 - added new feature\n",
"name": "1.1",
"synthetic": true,
"target": "396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52",
"target_type": "revision",
"target_url": "/api/1/revision/396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52/"
}
}


Revision artifact		Revision artifact
~~~~~~~~~~~~~~~~~		~~~~~~~~~~~~~~~~~

The metadata sent with the deposit is included in the revision which affects		The metadata sent with the deposit is stored in outside the revision,
		moraneggUnsubmitted Done Inline Actions delete `in` moranegg: delete `in`
the hash computation, thus resulting in a unique identifier.		and does not affect the hash computation.
		moraneggUnsubmitted Done Inline Actions can you add a phrase about `author` and `committer` in a revision? At the moment it is always SWH, since there is only a single person for each. moranegg: can you add a phrase about `author` and `committer` in a revision? At the moment it is always…
		vlorentzAuthorUnsubmitted Done Inline Actions D3941 vlorentz: D3941
This way, by depositing the same content with different metadata, will result
in two different revisions in the SWH archive.

The date mapping		The date mapping
^^^^^^^^^^^^^^^^		^^^^^^^^^^^^^^^^

A deposit may contain 4 different dates concerning the software artifacts.		A deposit may contain 4 different dates concerning the software artifacts.

The deposit's revision will reflect the most accurate point in time available.		The deposit's revision will reflect the most accurate point in time available.
Here are all dates that can be available in a deposit:		Here are all dates that can be available in a deposit:
▲ Show 20 Lines • Show All 222 Lines • ▼ Show 20 Lines	When the loading has failed, the deposit entry is updated:
- ``swh-id`` and ``complete_data`` remains as is		- ``swh-id`` and ``complete_data`` remains as is

Note: As a further improvement, we may prefer having a retry policy with		Note: As a further improvement, we may prefer having a retry policy with
graceful delays for further scheduling.		graceful delays for further scheduling.

Metadata loading		Metadata loading
~~~~~~~~~~~~~~~~		~~~~~~~~~~~~~~~~

- the metadata received with the deposit are kept in the `metadata` fields		- the metadata received with the deposit are kept in a dedicated table
of the revision and in the ```origin_metadata`` table to facilitate search		``raw_extrinsic_metadata``, distinct from the ``revision`` and ``origin``
over origin metadata.		tables.

- provider\_id and tool\_id are resolved by the prepare\_metadata method in the
loader-core

- the origin\_metadata entry is sent to storage by the send\_origin\_metadata		- ``authority`` is computed from the deposit client information, and ``fetcher``
in the loader-core		is the deposit loader.

origin\_metadata table:

::

id bigint PK
origin bigint
discovery_date date
provider_id bigint FK // (from provider table)
tool_id bigint FK // indexer_configuration_id tool used for extraction
metadata jsonb // before translation