Changeset View
Changeset View
Standalone View
Standalone View
docs/specs/spec-loading.rst
Show First 20 Lines • Show All 46 Lines • ▼ Show 20 Lines | |||||
| directory | root directory of the expanded submitted| | | directory | root directory of the expanded submitted| | ||||
| | tarball | | | | tarball | | ||||
+------------------------------------+-----------------------------------------+ | +------------------------------------+-----------------------------------------+ | ||||
Origin artifact | Origin artifact | ||||
~~~~~~~~~~~~~~~ | ~~~~~~~~~~~~~~~ | ||||
We create an origin using the url in the deposited metadata. | We create an origin URL by concatenating the client URI and the value of the | ||||
moranegg: maybe we should add an example from T2391? not sure it is the right place to do so.. | |||||
Done Inline ActionsWe can add a link to it once the documentation exists vlorentz: We can add a link to it once the documentation exists | |||||
The current deposit and future deposits with the same url or external_id | Slug header of the initial POST request of the deposit. | ||||
will be associated to this origin. | |||||
.. code-block:: json | .. code-block:: json | ||||
{ | { | ||||
"origin": { | "origin": { | ||||
"id": 89283768, | "id": 89283768, | ||||
"origin_visits_url": "/api/1/origin/89283768/visits/", | "origin_visits_url": "/api/1/origin/89283768/visits/", | ||||
"type": "deposit", | "type": "deposit", | ||||
Show All 32 Lines | .. code-block:: json | ||||
"visit": 1 | "visit": 1 | ||||
} | } | ||||
] | ] | ||||
} | } | ||||
Snapshot artifact | Snapshot artifact | ||||
~~~~~~~~~~~~~~~~~ | ~~~~~~~~~~~~~~~~~ | ||||
The snapshot represents one deposit push. The master branch points to a | The snapshot represents one deposit push. The ``HEAD`` branch points to a | ||||
synthetic revision. We will create a second branch pointing to a release | synthetic revision. | ||||
artifact, if the indicate that the deposit is a release with a `releaseNotes`. | |||||
.. code-block:: json | .. code-block:: json | ||||
{ | { | ||||
"snapshot": { | "snapshot": { | ||||
"branches": { | "branches": { | ||||
"master": { | "HEAD": { | ||||
"target": "396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52", | "target": "396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52", | ||||
"target_type": "revision", | "target_type": "revision", | ||||
"target_url": "/api/1/revision/396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52/" | "target_url": "/api/1/revision/396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52/" | ||||
} | } | ||||
"refs/tags/v1.1": { | |||||
"target": "a9f3396f372ed4a51d75e15ca16c1c2df1fc5c97", | |||||
"target_type": "release", | |||||
"target_url": "/api/1/release/a9f3396f372ed4a51d75e15ca16c1c2df1fc5c97/" | |||||
} | |||||
}, | }, | ||||
"id": "a3773941561cc557853898773a19c07cfe2efc5a", | "id": "a3773941561cc557853898773a19c07cfe2efc5a", | ||||
"next_branch": null | "next_branch": null | ||||
} | } | ||||
} | } | ||||
Note that previous versions of the deposit-loader named the branch ``master`` | |||||
instead, and created release branches under certain conditions. | |||||
Release artifact | Release artifact | ||||
~~~~~~~~~~~~~~~~ | ~~~~~~~~~~~~~~~~ | ||||
.. warning:: | |||||
This part of the specification is not implemented yet, only releases are | |||||
currently being created. | |||||
The content is deposited with a set of descriptive metadata in the CodeMeta | The content is deposited with a set of descriptive metadata in the CodeMeta | ||||
vocabulary. The following CodeMeta terms implies that the | vocabulary. The following CodeMeta terms implies that the | ||||
artifact is a release: | artifact is a release: | ||||
- `releaseNotes` | - `releaseNotes` | ||||
- `softwareVersion` | - `softwareVersion` | ||||
If present, a release artifact will be created with the mapping below: | If present, a release artifact will be created with the mapping below: | ||||
Show All 30 Lines | .. code-block:: json | ||||
"message": "AffectationRO Version 1.1 - added new feature\n", | "message": "AffectationRO Version 1.1 - added new feature\n", | ||||
"name": "1.1", | "name": "1.1", | ||||
"synthetic": true, | "synthetic": true, | ||||
"target": "396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52", | "target": "396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52", | ||||
"target_type": "revision", | "target_type": "revision", | ||||
"target_url": "/api/1/revision/396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52/" | "target_url": "/api/1/revision/396b1ff29f7c75a0a3cc36f30e24ff7bae70bb52/" | ||||
} | } | ||||
} | } | ||||
Done Inline ActionsWe didn't create, it was in spec but never implemented. We still need to implement this, which is described in T1755. I prefer keeping the release in spec and saying it is on the roadmap. moranegg: We didn't create, it was in spec but never implemented.
We still need to implement this, which… | |||||
Done Inline Actionsgood point vlorentz: good point | |||||
Revision artifact | Revision artifact | ||||
~~~~~~~~~~~~~~~~~ | ~~~~~~~~~~~~~~~~~ | ||||
The metadata sent with the deposit is included in the revision which affects | The metadata sent with the deposit is stored outside the revision, | ||||
Done Inline Actionscan you add a phrase about author and committer in a revision? moranegg: can you add a phrase about `author` and `committer` in a revision?
At the moment it is always… | |||||
Done Inline Actionsvlorentz: D3941 | |||||
the hash computation, thus resulting in a unique identifier. | and does not affect the hash computation. | ||||
Done Inline Actionsdelete in moranegg: delete `in` | |||||
This way, by depositing the same content with different metadata, will result | |||||
in two different revisions in the SWH archive. | |||||
The date mapping | The date mapping | ||||
^^^^^^^^^^^^^^^^ | ^^^^^^^^^^^^^^^^ | ||||
A deposit may contain 4 different dates concerning the software artifacts. | A deposit may contain 4 different dates concerning the software artifacts. | ||||
The deposit's revision will reflect the most accurate point in time available. | The deposit's revision will reflect the most accurate point in time available. | ||||
Here are all dates that can be available in a deposit: | Here are all dates that can be available in a deposit: | ||||
▲ Show 20 Lines • Show All 222 Lines • ▼ Show 20 Lines | When the loading has failed, the deposit entry is updated: | ||||
- ``swh-id`` and ``complete_data`` remains as is | - ``swh-id`` and ``complete_data`` remains as is | ||||
*Note:* As a further improvement, we may prefer having a retry policy with | *Note:* As a further improvement, we may prefer having a retry policy with | ||||
graceful delays for further scheduling. | graceful delays for further scheduling. | ||||
Metadata loading | Metadata loading | ||||
~~~~~~~~~~~~~~~~ | ~~~~~~~~~~~~~~~~ | ||||
- the metadata received with the deposit are kept in the `metadata` fields | - the metadata received with the deposit are kept in a dedicated table | ||||
of the revision and in the ```origin_metadata`` table to facilitate search | ``raw_extrinsic_metadata``, distinct from the ``revision`` and ``origin`` | ||||
over origin metadata. | tables. | ||||
- provider\_id and tool\_id are resolved by the prepare\_metadata method in the | - ``authority`` is computed from the deposit client information, and ``fetcher`` | ||||
loader-core | is the deposit loader. | ||||
- the origin\_metadata entry is sent to storage by the send\_origin\_metadata | |||||
in the loader-core | |||||
origin\_metadata table: | |||||
:: | |||||
id bigint PK | |||||
origin bigint | |||||
discovery_date date | |||||
provider_id bigint FK // (from provider table) | |||||
tool_id bigint FK // indexer_configuration_id tool used for extraction | |||||
metadata jsonb // before translation |
maybe we should add an example from T2391? not sure it is the right place to do so..