diff --git a/docs/README.rst b/docs/README.rst --- a/docs/README.rst +++ b/docs/README.rst @@ -52,18 +52,18 @@ - the deposit API uses the `SWORD v2`_ API, thus requires some tooling to send deposits to SWH. These tools are provided with this repository. -See the :ref:`user-manual` page for more details on how to use the deposit client +See the :ref:`deposit-user-manual` page for more details on how to use the deposit client command line tools to push a deposit in the SWH Archive. -See the :ref:`swh-api-specifications` reference pages of the SWORDv2 API implementation +See the :ref:`deposit-api-specifications` reference pages of the SWORDv2 API implementation in `swh.deposit` if you want to do upload deposits using HTTP requests. Read the :ref:`metadata` chapter to get more details on what metadata are supported when doing a deposit. -See :ref:`swh-deposit-dev` if you want to hack the code of the `swh.deposit` module. +See :ref:`swh-deposit-dev-env` if you want to hack the code of the `swh.deposit` module. -See :ref:`swh-deposit-deployment` if you want to deploy your own copy of the +See :ref:`swh-deposit-prod-env` if you want to deploy your own copy of the `swh.deposit` stack. diff --git a/docs/spec-api.rst b/docs/api/api-documentation.rst copy from docs/spec-api.rst copy to docs/api/api-documentation.rst diff --git a/docs/api/index.rst b/docs/api/index.rst new file mode 100644 --- /dev/null +++ b/docs/api/index.rst @@ -0,0 +1,13 @@ +.. _swh-deposit-api: + +Deposit API +=========== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + user-manual + api-documentation + metadata + use-cases diff --git a/docs/metadata.rst b/docs/api/metadata.rst copy from docs/metadata.rst copy to docs/api/metadata.rst diff --git a/docs/specs/blueprint.rst b/docs/api/use-cases.rst copy from docs/specs/blueprint.rst copy to docs/api/use-cases.rst --- a/docs/specs/blueprint.rst +++ b/docs/api/use-cases.rst @@ -1,3 +1,5 @@ +.. _deposit-use-cases: + Use cases ========= diff --git a/docs/user-manual.rst b/docs/api/user-manual.rst copy from docs/user-manual.rst copy to docs/api/user-manual.rst --- a/docs/user-manual.rst +++ b/docs/api/user-manual.rst @@ -1,4 +1,4 @@ -.. _user-manual: +.. _deposit-user-manual: User Manual =========== diff --git a/docs/cli.rst b/docs/cli.rst --- a/docs/cli.rst +++ b/docs/cli.rst @@ -17,6 +17,8 @@ :prog: swh deposit admin :nested: full +.. _swh-deposit-cli-client: + Deposit client tools -------------------- diff --git a/docs/index.rst b/docs/index.rst --- a/docs/index.rst +++ b/docs/index.rst @@ -6,13 +6,9 @@ :maxdepth: 2 :caption: Contents: - user-manual - metadata - spec-api - dev-info - sys-info + api/index + internals/index specs/index - tests/tests_HAL.rst Reference Documentation @@ -23,4 +19,3 @@ cli /apidoc/swh.deposit - authentication.rst diff --git a/docs/authentication.rst b/docs/internals/authentication.rst rename from docs/authentication.rst rename to docs/internals/authentication.rst diff --git a/docs/dev-info.rst b/docs/internals/dev-environment.rst rename from docs/dev-info.rst rename to docs/internals/dev-environment.rst --- a/docs/dev-info.rst +++ b/docs/internals/dev-environment.rst @@ -1,7 +1,7 @@ -.. _swh-deposit-dev: +.. _swh-deposit-dev-env: -Hacking on swh-deposit -====================== +Running swh-deposit locally +=========================== There are multiple modes to run and test the server locally: diff --git a/docs/internals/index.rst b/docs/internals/index.rst new file mode 100644 --- /dev/null +++ b/docs/internals/index.rst @@ -0,0 +1,14 @@ +.. _swh-deposit-internals: + +Deposit internals +================= + +This chapter describes how swh-deposit works internally, +and how to run it (either in production or locally for development). + +.. toctree:: + :maxdepth: 1 + + dev-environment + prod-environment + authentication diff --git a/docs/sys-info.rst b/docs/internals/prod-environment.rst rename from docs/sys-info.rst rename to docs/internals/prod-environment.rst --- a/docs/sys-info.rst +++ b/docs/internals/prod-environment.rst @@ -1,7 +1,7 @@ -.. _swh-deposit-deployment: +.. _swh-deposit-prod-env: -Deployment -========== +Production deployment +===================== The deposit is architectured around 3 parts: @@ -12,7 +12,7 @@ correct (the checker), another worker service dedicated to actually ingest the deposit into the swh archive. - - client: a python sčript `swh deposit` command line interface. + - client: a python script `swh deposit` command line interface. All those are packaged in 3 separated debian packages, created and uploaded to the swh debian repository. The deposit server and workers configuration are managed by puppet diff --git a/docs/metadata.rst b/docs/metadata.rst --- a/docs/metadata.rst +++ b/docs/metadata.rst @@ -1,200 +1,4 @@ -.. _deposit-metadata: +:orphan: -Deposit metadata -================ +This page was moved to: :ref:`deposit-metadata` -When making a software deposit into the SWH archive, one can add -information describing the software artifact and the software project. - - -.. _metadata-requirements: - -Metadata requirements ---------------------- - -- **the schema/vocabulary** used *MUST* be specified with a persistent url - (DublinCore, DOAP, CodeMeta, etc.) - - .. code:: xml - - - or - - or - - -- **the name** of the software deposit *MUST* be provided [atom:title, - codemeta:name, dcterms:title] - -- **the authors** of the software deposit *MUST* be provided - -- **the url** representing the location of the source *MAY* be provided under - the url tag. The url will be used for creating an origin object in the - archive. - - .. code:: xml - - http://example.com/my_project - -- **the create\_origin** tag *SHOULD* be used to specify the URL of the origin - to create (otherwise, a fallback is created using the slug, or a random - string if missing) - -- **the description** of the software deposit *SHOULD* be provided - [codemeta:description]: short or long description of the software - -- **the license/s** of the software - deposit *SHOULD* be provided [codemeta:license] - -- other metadata *MAY* be added with terms defined by the schema in use. - -Examples --------- - -Using only Atom -^^^^^^^^^^^^^^^ - -.. code:: xml - - - - Awesome Compiler - urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a - 2017-10-07T15:17:08Z - some awesome author - - - - - - - -Using Atom with CodeMeta -^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: xml - - - - Awesome Compiler - urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a - - - - - - 1785io25c695 - origin url - other identifier, DOI, ARK - Domain - - description - key-word 1 - key-word 2 - creation date - publication date - comment - - article name - article id - - - Collaboration/Projet - project name - id - - see also - Sponsor A - Sponsor B - Platform/OS - dependencies - Version - active - - license - url spdx - - .Net Framework 3.0 - Python2.3 - - author1 - Inria - UPMC - - - author2 - Inria - UPMC - - http://code.com - language 1 - language 2 - http://issuetracker.com - - -Using Atom with DublinCore and CodeMeta (multi-schema entry) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code:: xml - - - - Awesome Compiler - urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a - - - - - - hal-01587361 - doi:10.5281/zenodo.438684 - The assignment problem - AffectationRO - author - [INFO] Computer Science [cs] - [INFO.INFO-RO] Computer Science [cs]/Operations Research [cs.RO] - SOFTWARE - Project in OR: The assignment problemA java implementation for the assignment problem first release - description fr - 2015-06-01 - 2017-10-19 - en - - - origin url - - 1.0.0 - key word - Comment - Rfrence interne - - link - Sponsor - - Platform/OS - dependencies - Ended - - license - url spdx - - - http://code.com - language 1 - language 2 - - -Note ----- -We aim on harmonizing the metadata from different origins and thus -metadata will be translated to the `CodeMeta -v.2 `__ vocabulary if -possible. diff --git a/docs/spec-api.rst b/docs/spec-api.rst --- a/docs/spec-api.rst +++ b/docs/spec-api.rst @@ -1,115 +1,3 @@ -.. _swh-api-specifications: +:orphan: -API Documentation -================= - -This is `Software Heritage `__'s -`SWORD -2.0 `__ -Server implementation. - -**S.W.O.R.D** (**S**\ imple **W**\ eb-Service **O**\ ffering -**R**\ epository **D**\ eposit) is an interoperability standard for -digital file deposit. - -This implementation will permit interaction between a client (a repository) and -a server (SWH repository) to push deposits of software source code archives -with associated metadata. - -*Note:* - -* In the following document, we will use the ``archive`` or ``software source - code archive`` interchangeably. -* The supported archive formats are: - - * zip: common zip archive (no multi-disk zip files). - * tar: tar archive without compression or optionally any of the following - compression algorithm gzip (.tar.gz, .tgz), bzip2 (.tar.bz2) , or lzma - (.tar.lzma) - -.. _swh-deposit-collection: - -Collection ----------- - -SWORD defines a ``collection`` concept. In SWH's case, this collection -refers to a group of deposits. A ``deposit`` is some form of software -source code archive(s) associated with metadata. -By default the client's collection will have the client's name. - -Limitations ------------ -* upload limitation of 100Mib -* no mediation - -API overview ------------- - -API access is over HTTPS. - -The API is protected through basic authentication. - - -Endpoints ---------- - -The API endpoints are rooted at https://deposit.softwareheritage.org/1/. - -Data is sent and received as XML (as specified in the SWORD 2.0 -specification). - -.. include:: endpoints/service-document.rst - -.. include:: endpoints/collection.rst - -.. include:: endpoints/update-media.rst - -.. include:: endpoints/update-metadata.rst - -.. include:: endpoints/status.rst - -.. include:: endpoints/content.rst - - -Possible errors: ----------------- - -* common errors: - - * :http:statuscode:`401`:if a client does not provide credential or provide - wrong ones - * :http:statuscode:`403` a client tries access to a collection it does not own - * :http:statuscode:`404` if a client tries access to an unknown collection - * :http:statuscode:`404` if a client tries access to an unknown deposit - * :http:statuscode:`415` if a wrong media type is provided to the endpoint - -* archive/binary deposit: - - * :http:statuscode:`403` the length of the archive exceeds the max size - configured - * :http:statuscode:`412` the length or hash provided mismatch the reality of - the archive. - * :http:statuscode:`415` if a wrong media type is provided - -* multipart deposit: - - * :http:statuscode:`412` the md5 hash provided mismatch the reality of the - archive - * :http:statuscode:`415` if a wrong media type is provided - -* Atom entry deposit: - - * :http:statuscode:`400` if the request's body is empty (for creation only) - - - - -Sources -------- - -* `SWORD v2 specification - `__ -* `arxiv documentation `__ -* `Dataverse example `__ -* `SWORD used on HAL `__ -* `xml examples for CCSD `__ +This page was moved to: :ref:`deposit-api-specificationsa` diff --git a/docs/specs/blueprint.rst b/docs/specs/blueprint.rst --- a/docs/specs/blueprint.rst +++ b/docs/specs/blueprint.rst @@ -1,194 +1,3 @@ -Use cases -========= +:orphan: -The general idea is that a deposit can be created either in a single request -or by multiple requests to allow the user to add elements to the deposit piece -by piece (be it the deposited data or the metadata describing it). - -An update request that does not have the `In-Progress: true` HTTP header will -de facto declare the deposit as *completed* (aka in the `deposited` status; see -below) and thus ready for ingestion. - -Once the deposit is declared *complete* by the user, the server performs a few -validation checks. Then, if valid, schedule the ingestion of the deposited data -in the Software Heritage Archive (SWH). - -There is a `status` property attached to a deposit allowing to follow the -processing workflow of the deposit. For example, when this ingestion task -completes successfully, the deposit is marked as `done`. - - -Possible deposit statuses are: - -partial - The deposit is partially received, since it can be done in - multiple requests. - -expired - Deposit was there too long and is new deemed ready to be - garbage-collected. - -deposited - Deposit is complete, ready to be checked. - -rejected - Deposit failed the checks. - -verified - Deposit passed the checks and is ready for loading. - -loading - Injection is ongoing on SWH's side. - -done - Loading is successful. - -failed - Loading failed. - - -This document describes the possible scenarios for creating or updating a -deposit. - - -Deposit creation ----------------- - -From client's deposit repository server to SWH's repository server: - -1. The client requests for the server's abilities and its associated - :ref:`collections ` using the *SD/service document uri* - (:http:get:`/1/servicedocument/`). - -2. The server answers the client with the service document which lists the - *collections* linked to the user account (most of the time, there will one and - only one collection linked to the user's account). Each of these collection can - be used to push a deposit via its *COL/collection IRI*. - -3. The client sends a deposit (a zip archive, some metadata or both) through - the *COL/collection uri*. - - This can be done in: - - * one POST request (metadata + archive) without the `In-Progress: true` header: - - - :http:post:`/1/(str:collection-name)/` - - * one POST request (metadata or archive) **with** `In-Progress: true` header: - - - :http:post:`/1/(str:collection-name)/` - - plus one or more PUT or POST requests *to the update uris* - (*edit-media iri* or *edit iri*): - - - :http:post:`/1/(str:collection-name)/(int:deposit-id)/media/` - - :http:put:`/1/(str:collection-name)/(int:deposit-id)/media/` - - :http:post:`/1/(str:collection-name)/(int:deposit-id)/metadata/` - - :http:put:`/1/(str:collection-name)/(int:deposit-id)/metadata/` - - Then: - - a. Server validates the client's input or returns detailed error if any. - - b. Server stores information received (metadata or software archive source - code or both). - -4. The server notifies the client it acknowledged the client's request. An - ``http 201 Created`` response with a deposit receipt in the body response is - sent back. That deposit receipt will hold the necessary information to - eventually complete the deposit later on if it was incomplete (also known as - status ``partial``). - -Schema representation -^^^^^^^^^^^^^^^^^^^^^ - -Scenario: pushing a deposit via the SWORDv2_ protocol (nominal scenario): - -.. figure:: ../images/deposit-create-chart.svg - :alt: - - -Updating an existing deposit -"""""""""""""""""""""""""""" - -5. Client updates existing deposit through the *update uris* (one or more POST - or PUT requests to either the *edit-media iri* or *edit iri*). - - 1. Server validates the client's input or returns detailed error if any - - 2. Server stores information received (metadata or software archive source - code or both) - -This would be the case for example if the client initially posted a -``partial`` deposit (e.g. only metadata with no archive, or an archive -without metadata, or a split archive because the initial one exceeded -the limit size imposed by swh repository deposit). - -The content of a deposit can only be updated while it is in the ``partial`` -state; this causes the content to be **replaced** (the old version is discarded). - -Its metadata, however, can also be updated while in the ``done`` state; -which adds a new version of the metadata in the SWH archive, -**in addition to** the old one(s). -In this state, ``In-Progress`` is not allowed, so the deposit cannot go back -in the ``partial`` state, but only to ``deposited``. -As a failsafe, to avoid accidentally updating the wrong deposit, this requires -the ``X-Check-SWHID`` HTTP header to be set to the value of the SWHID of the -deposit's content (returned after the deposit finished loading). - - -Schema representation -""""""""""""""""""""" - -Scenario: updating a deposit via SWORDv2_ protocol: - -.. figure:: ../images/deposit-update-chart.svg - :alt: - - -Deleting deposit (or associated archive, or associated metadata) -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -6. Deposit deletion is possible as long as the deposit is still in ``partial`` - state. - - 1. Server validates the client's input or returns detailed error if any - 2. Server actually delete information according to request - - -Schema representation -^^^^^^^^^^^^^^^^^^^^^ - -Scenario: deleting a deposit via SWORDv2_ protocol: - -.. figure:: ../images/deposit-delete-chart.svg - :alt: - - -Client asks for operation status -"""""""""""""""""""""""""""""""" - -7. Operation status can be read through a GET query to the *state iri*. - - -Server: Triggering deposit checks -""""""""""""""""""""""""""""""""" - -Once the status ``deposited`` is reached for a deposit, checks for the -associated archive(s) and metadata will be triggered. If those checks -fail, the status is changed to ``rejected`` and nothing more happens -there. Otherwise, the status is changed to ``verified``. - - -Server: Triggering deposit load -""""""""""""""""""""""""""""""" - -Once the status ``verified`` is reached for a deposit, loading the -deposit with its associated metadata will be triggered. - -The loading will result on status update, either ``done`` or ``failed`` -(depending on the loading's status). - -This is described in the :ref:`loading specifications document `. - -.. _SWORDv2: http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html +This page was moved to: :ref:`deposit-use-cases` diff --git a/docs/specs/index.rst b/docs/specs/index.rst --- a/docs/specs/index.rst +++ b/docs/specs/index.rst @@ -1,13 +1,12 @@ .. _swh-deposit-specs: -Blueprint Specifications -========================= +Specifications +============== .. toctree:: :maxdepth: 1 :caption: Contents: - blueprint.rst spec-loading.rst protocol-reference.rst spec-meta-deposit.rst diff --git a/docs/specs/spec-loading.rst b/docs/specs/spec-loading.rst --- a/docs/specs/spec-loading.rst +++ b/docs/specs/spec-loading.rst @@ -154,7 +154,7 @@ .. warning:: - This part of the specification is not implemented yet, only releases are + This part of the specification is not implemented yet, only revisions are currently being created. The content is deposited with a set of descriptive metadata in the CodeMeta diff --git a/docs/tests/tests_HAL.rst b/docs/tests/tests_HAL.rst deleted file mode 100644 --- a/docs/tests/tests_HAL.rst +++ /dev/null @@ -1,67 +0,0 @@ -Tests scenarios for client -========================== - -Scenarios for HAL- on HAL's platform ------------------------------------- - -The same procedure is used for all tests: - -Software Author: - -#. prepare content -#. fill out form -#. submit - -HAL moderator: - -#. review content submitted -#. check metadata fields on HAL -#. validate submission - -SWH side: - -1. check content in SWH: - - - directory was created - - revision was created - - release was created when releaseNotes and softwareVersion was included (new feature!) - - origin corresponds to HAL url - -2. check metadata fields on SWH (in revision) -3. check directory -4. check swh-id on HAL -5. check browsability when entering SWH artifact from HAL -6. check vault artifact recreation -7. access deposit's origin from SWH - -+-------------+-------------------------------------------+----------+---------+-----------------------------------------+ -| scenario | test case | data | result | exceptions or specific checks | -+=============+===========================================+==========+=========+=========================================+ -| submit code | content: .tar.gz | .zip | success | | -+-------------+-------------------------------------------+----------+---------+-----------------------------------------+ -| submit code | content: .zip | .tar.gz | success | | -+-------------+-------------------------------------------+----------+---------+-----------------------------------------+ -| submit code | content: no content | empty | fail | blocked on HAL | -+-------------+-------------------------------------------+----------+---------+-----------------------------------------+ -| submit code | content: double compression (.zip in .zip)| .zip x 2 | fail | status `failed` on SWH | -+-------------+-------------------------------------------+----------+---------+-----------------------------------------+ -| submit code | all metadata-single entry | metadata | success | check that all metadata is transmitted | -+-------------+-------------------------------------------+----------+---------+-----------------------------------------+ -| submit code | multiple entries | metadata | success | languages / authors / descriptions | -+-------------+-------------------------------------------+----------+---------+-----------------------------------------+ -| new version | new content- same metadata | content | success | check new swh-id in SWH and HAL | -+-------------+-------------------------------------------+----------+---------+-----------------------------------------+ -| new version | same content- new metadata | metadata | ? | dead angle- doesn't arrives to SWH | -+-------------+-------------------------------------------+----------+---------+-----------------------------------------+ -| new version | new content-new metadata | C & M | success | check artifacts history in revisions | -+-------------+-------------------------------------------+----------+---------+-----------------------------------------+ -| submit code | deposit on another hal platform | C & M | success | | -+-------------+-------------------------------------------+----------+---------+-----------------------------------------+ - -Past known bugs: - -- v2 problem, where swh-id from first version is kept in the second version - instead of the new swh-id. -- when deposit workers are down- error 500 is returned on HAL without real - explanation (because there is no error on SWH- deposit status - stays `deposited`). diff --git a/docs/user-manual.rst b/docs/user-manual.rst --- a/docs/user-manual.rst +++ b/docs/user-manual.rst @@ -1,486 +1,4 @@ -.. _user-manual: +:orphan: -User Manual -=========== +This page was moved to: :ref:`deposit-user-manual` -This is a guide for how to prepare and push a software deposit with -the `swh deposit` commands. - - -Requirements ------------- - -You need to have an account on the Software Heritage deposit application to be -able to use the service. - -Please `contact the Software Heritage team `_ for -more information on how to get access to this service. - -For testing purpose, a test instance `is available -`_ [#f1]_ and will be used in the examples below. - -Once you have an account, you should get a set of access credentials as a -`login` and a `password` (identified as ```` and ```` in the -remaining of this document). A deposit account also comes with a "provider URL" -which is used by SWH to build the :term:`Origin URL` of deposits -created using this account. - - -Installation ------------- - -To install the `swh.deposit` command line tools, you need a working Python 3.7+ -environment. It is strongly recommended you use a `virtualenv -`_ for this. - -.. code:: console - - $ python3 -m virtualenv deposit - [...] - $ source deposit/bin/activate - (deposit)$ pip install swh.deposit - [...] - (deposit)$ swh deposit --help - Usage: swh deposit [OPTIONS] COMMAND [ARGS]... - - Deposit main command - - Options: - -h, --help Show this message and exit. - - Commands: - admin Server administration tasks (manipulate user or... - status Deposit's status - upload Software Heritage Public Deposit Client Create/Update... - (deposit)$ - -Note: in the examples below, we use the `jq`_ tool to make json outputs nicer. -If you do have it already, you may install it using your distribution's -packaging system. For example, on a Debian system: - -.. _jq: https://stedolan.github.io/jq/ - -.. code:: console - - $ sudo apt install jq - -.. _prepare_deposit - -Prepare a deposit ------------------ - -* compress the files in a supported archive format: - - - zip: common zip archive (no multi-disk zip files). - - tar: tar archive without compression or optionally any of the - following compression algorithm gzip (`.tar.gz`, `.tgz`), bzip2 - (`.tar.bz2`) , or lzma (`.tar.lzma`) - -* (Optional) prepare a metadata file (more details :ref:`deposit-metadata`): - -Example: - -Assuming you want to deposit the source code of `belenios -`_ version 1.12 - -.. code:: console - - (deposit)$ wget https://gitlab.inria.fr/belenios/belenios/-/archive/1.12/belenios-1.12.zip - [...] - 2020-10-28 11:40:37 (4,56 MB/s) - ‘belenios-1.12.zip’ saved [449880/449880] - (deposit)$ - -Then you need to prepare a metadata file allowing you to give detailed -information on your deposited source code. A rather minimal Atom with Codemeta -file could be: - -.. code:: console - - (deposit)$ cat metadata.xml - - - Verifiable online voting system - belenios-01243065 - https://gitlab.inria.fr/belenios/belenios - test - Online voting - Verifiable online voting system - 1.12 - opam - stable - ocaml - - GNU Affero General Public License - - - Belenios - belenios@example.com - - - Belenios Test User - - - - - - - - - (deposit)$ - -Please read the :ref:`deposit-metadata` page for a more detailed view on the -metadata file formats and semantics. - - -Push a deposit --------------- - -You can push a deposit with: - -* a single deposit (archive + metadata): - - The user posts in one query a software - source code archive and associated metadata. - The deposit is directly marked with status ``deposited``. - -* a multisteps deposit: - - 1. Create an incomplete deposit (marked with status ``partial``) - 2. Add data to a deposit (in multiple requests if needed) - 3. Finalize deposit (the status becomes ``deposited``) - -* a metadata-only deposit: - - The user posts in one query an associated metadata file on a :ref:`SWHID - ` object. The deposit is directly marked with status - ``done``. - -Overall, a deposit can be a in series of steps as follow: - -.. figure:: images/status.svg - :alt: - -The important things to notice for now is that it can be: - -partial: - the deposit is partially received - -expired: - deposit has been there too long and is now deemed - ready to be garbage collected - -deposited: - deposit is complete and is ready to be checked to ensure data consistency - -verified: - deposit is fully received, checked, and ready for loading - -loading: - loading is ongoing on swh's side - -done: - loading is successful - -failed: - loading is a failure - - -When you push a deposit, it is either in the `deposited` state or in the -`partial` state if you asked for a partial upload. - - - -Single deposit -^^^^^^^^^^^^^^ - -Once the files are ready for deposit, we want to do the actual deposit in one -shot, i.e. sending both the archive (zip) file and the metadata file. - -* 1 archive (content-type ``application/zip`` or ``application/x-tar``) -* 1 metadata file in atom xml format (``content-type: application/atom+xml;type=entry``) - -For this, we need to provide the: - -* arguments: ``--username 'name' --password 'pass'`` as credentials -* archive's path (example: ``--archive path/to/archive-name.tgz``) -* metadata file path (example: ``--metadata path/to/metadata.xml``) - -to the `swh deposit upload` command. - - - -Example: - -To push the Belenios 1.12 we prepared previously on the testing instance of the -deposit: - -.. code:: console - - (deposit)$ ls - belenios-1.12.zip metadata.xml deposit - (deposit)$ swh deposit upload --username --password \ - --url https://deposit.staging.swh.network/1 \ - --slug belenios-01243065 \ - --archive belenios.zip \ - --metadata metadata.xml \ - --format json | jq - { - 'deposit_status': 'deposited', - 'deposit_id': '1', - 'deposit_date': 'Oct. 28, 2020, 1:52 p.m.', - 'deposit_status_detail': None - } - - (deposit)$ - - -You just posted a deposit to your main collection on Software Heritage (staging -area)! - -The returned value is a JSON dict, in which you will notably find the deposit -id (needed to check for its status later on) and the current status, which -should be `deposited` if no error has occurred. - -Note: As the deposit is in ``deposited`` status, you can no longer -update the deposit after this query. It will be answered with a 403 -(Forbidden) answer. - -If something went wrong, an equivalent response will be given with the -`error` and `detail` keys explaining the issue, e.g.: - -.. code:: console - - { - 'error': 'Unknown collection name xyz', - 'detail': None, - 'deposit_status': None, - 'deposit_status_detail': None, - 'deposit_swh_id': None, - 'status': 404 - } - - -Once the deposit has been done, you can check its status using the `swh deposit -status` command: - -.. code:: console - - (deposit)$ swh deposit status --username --password \ - --url https://deposit.staging.swh.network/1 \ - --deposit-id 1 -f json | jq - { - "deposit_id": "1", - "deposit_status": "done", - "deposit_status_detail": "The deposit has been successfully loaded into the Software Heritage archive", - "deposit_swh_id": "swh:1:dir:63a6fc0ed8f69bf66ccbf99fc0472e30ef0a895a", - "deposit_swh_id_context": "swh:1:dir:63a6fc0ed8f69bf66ccbf99fc0472e30ef0a895a;origin=https://softwareheritage.org/belenios-01234065;visit=swh:1:snp:0ae536667689da7047bfb7aa9f37f5958e9f4647;anchor=swh:1:rev:17ad98c940104d45b6b6bd6fba9aa832eeb95638;path=/", - "deposit_external_id": "belenios-01234065" - } - - -Metadata-only deposit -^^^^^^^^^^^^^^^^^^^^^ - -This allows to deposit only metadata information on a :ref:`SWHID reference -`. Prepare a metadata file as described in the -:ref:`prepare deposit section ` - -Ensure this metadata file also declares a :ref:`SWHID reference -`: - -.. code:: xml - - - - - - - - - - - - - - - -For this, we then need to provide the following information: - -* arguments: ``--username 'name' --password 'pass'`` as credentials -* metadata file path (example: ``--metadata path/to/metadata.xml``) - -to the `swh deposit metadata-only` command. - - -Example: - -.. code:: console - - (deposit) swh deposit metadata-only --username --password \ - --url https://deposit.staging.swh.network/1 \ - --metadata ../deposit-swh.metadata-only.xml \ - --format json | jq . - { - "deposit_id": "29", - "deposit_status": "done", - "deposit_date": "Dec. 15, 2020, 11:37 a.m." - } - -For details on the metadata-only deposit, see the -:ref:`metadata-only deposit protocol reference ` - -Multisteps deposit -^^^^^^^^^^^^^^^^^^ - -In this case, the deposit is created by several requests, uploading objects -piece by piece. The steps to create a multisteps deposit: - -1. Create an partial deposit -"""""""""""""""""""""""""""" - -First use the ``--partial`` argument to declare there is more to come - -.. code:: console - - $ swh deposit upload --username name --password secret \ - --archive foo.tar.gz \ - --partial - - -2. Add content or metadata to the deposit -""""""""""""""""""""""""""""""""""""""""" - -Continue the deposit by using the ``--deposit-id`` argument given as a response -for the first step. You can continue adding content or metadata while you use -the ``--partial`` argument. - -To only add one new archive to the deposit: - -.. code:: console - - $ swh deposit upload --username name --password secret \ - --archive add-foo.tar.gz \ - --deposit-id 42 \ - --partial - -To only add metadata to the deposit: - -.. code:: console - - $ swh deposit upload --username name --password secret \ - --metadata add-foo.tar.gz.metadata.xml \ - --deposit-id 42 \ - --partial - - -3. Finalize deposit -""""""""""""""""""" - -On your last addition (same command as before), by not declaring it -``--partial``, the deposit will be considered completed. Its status will be -changed to ``deposited``: - -.. code:: console - - $ swh deposit upload --username name --password secret \ - --metadata add-foo.tar.gz.metadata.xml \ - --deposit-id 42 - - -Update deposit --------------- - -* Update deposit metadata: - - - only possible if the deposit status is ``done``, ``--deposit-id `` and - ``--swhid `` are provided - - - by using the ``--metadata`` flag, a path to an xml file - -.. code:: console - - $ swh deposit upload \ - --username name --password secret \ - --deposit-id 11 \ - --swhid swh:1:dir:2ddb1f0122c57c8479c28ba2fc973d18508e6420 \ - --metadata ../deposit-swh.update-metadata.xml - -* Replace deposit: - - - only possible if the deposit status is ``partial`` and - ``--deposit-id `` is provided - - - by using the ``--replace`` flag - - - ``--metadata-deposit`` replaces associated existing metadata - - ``--archive-deposit`` replaces associated archive(s) - - by default, with no flag or both, you'll replace associated - metadata and archive(s): - -.. code:: console - - $ swh deposit upload --username name --password secret \ - --deposit-id 11 \ - --archive updated-je-suis-gpl.tgz \ - --replace - -* Update a loaded deposit with a new version (this creates a new deposit): - - - by using the external-id with the ``--slug`` argument, you will - link the new deposit with its parent deposit: - -.. code:: console - - $ swh deposit upload --username name --password secret \ - --archive je-suis-gpl-v2.tgz \ - --slug 'je-suis-gpl' - - -Check the deposit's status --------------------------- - -You can check the status of the deposit by using the ``--deposit-id`` argument: - -.. code:: console - - $ swh deposit status --username name --password secret \ - --deposit-id 11 - -.. code:: json - - { - "deposit_id": 11, - "deposit_status": "deposited", - "deposit_swh_id": null, - "deposit_status_detail": "Deposit is ready for additional checks \ - (tarball ok, metadata, etc...)" - } - -When the deposit has been loaded into the archive, the status will be -marked ``done``. In the response, will also be available the -, . For example: - -.. code:: json - - { - "deposit_id": 11, - "deposit_status": "done", - "deposit_swh_id": "swh:1:dir:d83b7dda887dc790f7207608474650d4344b8df9", - "deposit_swh_id_context": "swh:1:dir:d83b7dda887dc790f7207608474650d4344b8df9;\ - origin=https://forge.softwareheritage.org/source/jesuisgpl/;\ - visit=swh:1:snp:68c0d26104d47e278dd6be07ed61fafb561d0d20;\ - anchor=swh:1:rev:e76ea49c9ffbb7f73611087ba6e999b19e5d71eb;path=/", - "deposit_status_detail": "The deposit has been successfully \ - loaded into the Software Heritage archive" - } - - - -.. rubric:: Footnotes - -.. [#f1] the test instance of the deposit is not yet available to external users, - but it should be available soon.