Page Menu
Home
Software Heritage
Search
Configure Global Search
Log In
Files
F7066381
D4430.id15686.diff
No One
Temporary
Actions
View File
Edit File
Delete File
View Transforms
Subscribe
Mute Notifications
Award Token
Flag For Later
Size
23 KB
Subscribers
None
D4430.id15686.diff
View Options
diff --git a/docs/getting-started.rst b/docs/getting-started.rst
deleted file mode 100644
--- a/docs/getting-started.rst
+++ /dev/null
@@ -1,285 +0,0 @@
-Getting Started
-===============
-
-This is a guide for how to prepare and push a software deposit with
-the `swh deposit` commands.
-
-The API is rooted at https://deposit.softwareheritage.org/1.
-
-For more details, see the `main documentation <./index.html>`__.
-
-Requirements
-------------
-
-You need to be referenced on SWH's client list to have:
-
-* credentials (needed for the basic authentication step)
-
- - in this document we reference ``<name>`` as the client's name and
- ``<pass>`` as its associated authentication password.
-
-* an associated collection_.
-
-
-.. _collection: https://bitworking.org/projects/atom/rfc5023#rfc.section.8.3.3
-
-
-`Contact us for more information.
-<https://www.softwareheritage.org/contact/>`__
-
-Prepare a deposit
------------------
-* compress the files in a supported archive format:
-
- - zip: common zip archive (no multi-disk zip files).
- - tar: tar archive without compression or optionally any of the
- following compression algorithm gzip (`.tar.gz`, `.tgz`), bzip2
- (`.tar.bz2`) , or lzma (`.tar.lzma`)
-
-* (Optional) prepare a metadata file (more details :ref:`deposit-metadata`):
-
-
-Push deposit
-------------
-You can push a deposit with:
-
-* a single deposit (archive + metadata):
-
- The user posts in one query a software
- source code archive and associated metadata.
- The deposit is directly marked with status ``deposited``.
-
-* a multisteps deposit:
-
- 1. Create an incomplete deposit (marked with status ``partial``)
- 2. Add data to a deposit (in multiple requests if needed)
- 3. Finalize deposit (the status becomes ``deposited``)
-
-
-Single deposit
-^^^^^^^^^^^^^^
-
-
-Once the files are ready for deposit, we want to do the actual deposit
-in one shot, sending exactly one POST query:
-
-* 1 archive (content-type ``application/zip`` or ``application/x-tar``)
-* 1 metadata file in atom xml format (``content-type: application/atom+xml;type=entry``)
-
-For this, we need to provide the:
-
-* arguments: ``--username 'name' --password 'pass'`` as credentials
-* archive's path (example: ``--archive path/to/archive-name.tgz``)
-* software's name (optional if a metadata filepath is specified and the
- artifact's name is included in the metadata file).
-* author's name (optional if a metadata filepath is specified and the authors
- are included in the metadata file). This can be specified multiple times in
- case of multiple authors.
-* (optionally) metadata file's path ``--metadata
- path/to/file.metadata.xml``.
-* (optionally) ``--slug 'your-id'`` argument, a reference to a unique identifier
- the client uses for the software object. If not provided, A UUID will be
- generated by SWH.
-
-You can do this with the following command:
-
-minimal deposit
-
-.. code:: shell
-
- $ swh deposit upload --username name --password secret \
- --author "Jane Doe" \
- --author "John Doe" \
- --name 'je-suis-gpl' \
- --archive je-suis-gpl.tgz
-
-with client's external identifier (``slug``)
-
-.. code:: shell
-
- $ swh deposit upload --username name --password secret \
- --author "Jane Doe" \
- --name 'je-suis-gpl' \
- --archive je-suis-gpl.tgz \
- --slug je-suis-gpl
-
-to a specific client's collection
-
-.. code:: shell
-
- $ swh deposit upload --username name --password secret \
- --author "Jane Doe" \
- --name 'je-suis-gpl' \
- --archive je-suis-gpl.tgz \
- --collection 'second-collection'
-
-
-You just posted a deposit to your collection on Software Heritage
-
-
-If everything went well, the successful response will contain the
-elements below:
-
-.. code:: shell
-
- {
- 'deposit_status': 'deposited',
- 'deposit_id': '7',
- 'deposit_date': 'Jan. 29, 2018, 12:29 p.m.'
- }
-
-Note: As the deposit is in ``deposited`` status, you can no longer
-update the deposit after this query. It will be answered with a 403
-forbidden answer.
-
-If something went wrong, an equivalent response will be given with the
-`error` and `detail` keys explaining the issue, e.g.:
-
-.. code:: shell
-
- {
- 'error': 'Unknown collection name xyz',
- 'detail': None,
- 'deposit_status': None,
- 'deposit_status_detail': None,
- 'deposit_swh_id': None,
- 'status': 404
- }
-
-
-
-multisteps deposit
-^^^^^^^^^^^^^^^^^^^^^^^^^
-The steps to create a multisteps deposit:
-
-1. Create an incomplete deposit
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-First use the ``--partial`` argument to declare there is more to come
-
-.. code:: shell
-
- $ swh deposit upload --username name --password secret \
- --archive foo.tar.gz \
- --partial
-
-
-2. Add content or metadata to the deposit
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Continue the deposit by using the ``--deposit-id`` argument given as a response
-for the first step. You can continue adding content or metadata while you use
-the ``--partial`` argument.
-
-To only add one new archive to the deposit:
-
-.. code:: shell
-
- $ swh deposit upload --username name --password secret \
- --archive add-foo.tar.gz \
- --deposit-id 42 \
- --partial
-
-To only add metadata to the deposit:
-
-.. code:: shell
-
- $ swh deposit upload --username name --password secret \
- --metadata add-foo.tar.gz.metadata.xml \
- --deposit-id 42 \
- --partial
-
-or:
-
-.. code:: shell
-
- $ swh deposit upload --username name --password secret \
- --name 'add-foo' --author 'someone' \
- --deposit-id 42 \
- --partial
-
-
-3. Finalize deposit
-~~~~~~~~~~~~~~~~~~~
-
-On your last addition (same command as before), by not declaring it
-``--partial``, the deposit will be considered completed. Its status will be
-changed to ``deposited``
-
-
-Update deposit
-----------------
-* replace deposit:
-
- - only possible if the deposit status is ``partial`` and
- ``--deposit-id <id>`` is provided
-
- - by using the ``--replace`` flag
-
- - ``--metadata-deposit`` replaces associated existing metadata
- - ``--archive-deposit`` replaces associated archive(s)
- - by default, with no flag or both, you'll replace associated
- metadata and archive(s):
-
-.. code:: shell
-
- $ swh deposit upload --username name --password secret \
- --deposit-id 11 \
- --archive updated-je-suis-gpl.tgz \
- --replace
-
-* update a loaded deposit with a new version:
-
- - by using the external-id with the ``--slug`` argument, you will
- link the new deposit with its parent deposit:
-
-.. code:: shell
-
- $ swh deposit upload --username name --password secret \
- --archive je-suis-gpl-v2.tgz \
- --slug 'je-suis-gpl' \
-
-
-
-Check the deposit's status
---------------------------
-
-You can check the status of the deposit by using the ``--deposit-id`` argument:
-
-.. code:: shell
-
- $ swh deposit status --username name --password secret \
- --deposit-id 11
-
-.. code:: json
-
- {
- 'deposit_id': '11',
- 'deposit_status': 'deposited',
- 'deposit_swh_id': None,
- 'deposit_status_detail': 'Deposit is ready for additional checks \
- (tarball ok, metadata, etc...)'
- }
-
-The different statuses:
-
-- **partial**: multipart deposit is still ongoing
-- **deposited**: deposit completed
-- **rejected**: deposit failed the checks
-- **verified**: content and metadata verified
-- **loading**: loading in-progress
-- **done**: loading completed successfully
-- **failed**: the deposit loading has failed
-
-When the deposit has been loaded into the archive, the status will be
-marked ``done``. In the response, will also be available the
-<deposit_swh_id>, <deposit_swh_id_context>. For example:
-
-.. code:: json
-
- {
- 'deposit_id': '11',
- 'deposit_status': 'done',
- 'deposit_swh_id': 'swh:1:dir:d83b7dda887dc790f7207608474650d4344b8df9',
- 'deposit_swh_id_context': 'swh:1:dir:d83b7dda887dc790f7207608474650d4344b8df9;origin=https://forge.softwareheritage.org/source/jesuisgpl/;visit=swh:1:snp:68c0d26104d47e278dd6be07ed61fafb561d0d20;anchor=swh:1:rev:e76ea49c9ffbb7f73611087ba6e999b19e5d71eb;path=/',
- 'deposit_status_detail': 'The deposit has been successfully \
- loaded into the Software Heritage archive'
- }
diff --git a/docs/index.rst b/docs/index.rst
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -52,7 +52,8 @@
- the deposit API uses the `SWORD v2`_ API, thus requires some tooling to send
deposits to SWH. These tools are provided with this repository.
-See the :ref:`getting-started` page for more details on how to use the deposit
+
+See the :ref:`user-manual` page for more details on how to use the deposit
client tools to push a deposit in the SWH Archive.
.. _codemeta: https://codemeta.github.io/
@@ -62,7 +63,7 @@
:maxdepth: 2
:caption: Contents:
- getting-started
+ user-manual
spec-api
metadata
dev-info
diff --git a/docs/user-manual.rst b/docs/user-manual.rst
new file mode 100644
--- /dev/null
+++ b/docs/user-manual.rst
@@ -0,0 +1,454 @@
+.. _user-manual:
+
+User Manual
+===========
+
+This is a guide for how to prepare and push a software deposit with
+the `swh deposit` commands.
+
+
+Requirements
+------------
+
+You need to have an account on the Software Heritage deposit application to be
+able to use the service.
+
+Please `contact the Software Heritage team
+<https://www.softwareheritage.org/contact/>`_ for more information on how to
+get access to this service.
+
+For testing purpose, a test instance `is available
+<https://deposit.staging.swh.network>`_ and will be used in the examples below.
+
+Once you have an account, you should get a set of access credentials as a
+`login` and a `password` (identified as ``<name>`` and ``<pass>`` in the
+remaining of this document), plus a XXX.
+
+Note that this deposit account
+
+Installation
+------------
+
+To install the `swh.deposit` command line tools, you need a working Python 3.7+
+environment. It is strongly recommended you use a `virtualenv
+<https://virtualenv.pypa.io/en/stable/>`_ for this.
+
+.. code:: console
+
+ $ python3 -m virtualenv deposit
+ [...]
+ $ source deposit/bin/activate
+ (deposit)$ pip install swh.deposit
+ [...]
+ (deposit)$ swh deposit --help
+ Usage: swh deposit [OPTIONS] COMMAND [ARGS]...
+
+ Deposit main command
+
+ Options:
+ -h, --help Show this message and exit.
+
+ Commands:
+ admin Server administration tasks (manipulate user or...
+ status Deposit's status
+ upload Software Heritage Public Deposit Client Create/Update...
+ (deposit)$
+
+Note: in the examples below, we use the `jq`_ tool to make json outputs nicer.
+If you do have it already, you may install it using your distribution's
+packaging system. Foe example, on a Debian system:
+
+.. _jq: https://stedolan.github.io/jq/
+
+.. code:: console
+
+ $ sudo apt install jq
+ [...]
+ $ jq
+ jq - commandline JSON processor [version 1.6]
+
+ Usage: jq [options] <jq filter> [file...]
+ jq [options] --args <jq filter> [strings...]
+ jq [options] --jsonargs <jq filter> [JSON_TEXTS...]
+
+ jq is a tool for processing JSON inputs, applying the given filter to
+ its JSON text inputs and producing the filter's results as JSON on
+ standard output.
+
+ The simplest filter is ., which copies jq's input to its output
+ unmodified (except for formatting, but note that IEEE754 is used
+ for number representation internally, with all that that implies).
+
+ For more advanced filters see the jq(1) manpage ("man jq")
+ and/or https://stedolan.github.io/jq
+
+ Example:
+
+ $ echo '{"foo": 0}' | jq .
+ {
+ "foo": 0
+ }
+
+
+Prepare a deposit
+-----------------
+
+* compress the files in a supported archive format:
+
+ - zip: common zip archive (no multi-disk zip files).
+ - tar: tar archive without compression or optionally any of the
+ following compression algorithm gzip (`.tar.gz`, `.tgz`), bzip2
+ (`.tar.bz2`) , or lzma (`.tar.lzma`)
+
+* (Optional) prepare a metadata file (more details :ref:`deposit-metadata`):
+
+Example:
+
+Assuming you want to deposit the source code of `belenios
+<https://gitlab.inria.fr/belenios/belenios>`_ version 1.12
+
+.. code:: console
+
+ (deposit)$ wget https://gitlab.inria.fr/belenios/belenios/-/archive/1.12/belenios-1.12.zip
+ [...]
+ 2020-10-28 11:40:37 (4,56 MB/s) - ‘belenios-1.12.zip’ saved [449880/449880]
+ (deposit)$
+
+Then, you can directly upload that zip file without giving explicitly a
+metadata file. In this case, a few options must be passed to the command to
+fill a basic metadata template that will be attached to the upload (see
+examples below).
+
+Then you need to prepare a metadata file allowing you to give detailed
+information on your deposited source code. A rather minimal Atom with Codemeta
+file could be:
+
+.. code:: console
+
+ (deposit)$ cat metadata.xml
+ <?xml version="1.0" encoding="utf-8"?>
+ <entry xmlns="http://www.w3.org/2005/Atom"
+ xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0">
+ <title>Verifiable online voting system</title>
+ <client>belenios</client>
+ <id>belenios-01243065</id>
+ <external_identifier>test-01243065</external_identifier>
+ <codemeta:url>https://gitlab.inria.fr/belenios/belenios</codemeta:url>
+ <codemeta:applicationCategory>test</codemeta:applicationCategory>
+ <codemeta:keywords>Online voting</codemeta:keywords>
+ <codemeta:description>Verifiable online voting system</codemeta:description>
+ <codemeta:version>1.12</codemeta:version>
+ <codemeta:runtimePlatform>opam</codemeta:runtimePlatform>
+ <codemeta:developmentStatus>stable</codemeta:developmentStatus>
+ <codemeta:programmingLanguage>ocaml</codemeta:programmingLanguage>
+ <codemeta:license>
+ <codemeta:name>GNU Affero General Public License</codemeta:name>
+ </codemeta:license>
+ <author>
+ <name>Belenios</name>
+ <email>belenios@example.com</email>
+ </author>
+ <codemeta:author>
+ <codemeta:name>Belenios Test User</codemeta:name>
+ </codemeta:author>
+ </entry>
+
+ (deposit)$
+
+Most of the fields in the example above are pretty self-explanatory, but some
+can be explained a bit:
+
+- `id`: XXX
+- `external_identifier`: XXX
+- `client`: XXX
+- `codemeta:url`: XXX
+
+Please see the :ref:`deposit-metadata` page for a more detailed view on the
+metadata file formats and semantics.
+
+
+Push a deposit
+--------------
+You can push a deposit with:
+
+* a single deposit (archive + metadata):
+
+ The user posts in one query a software
+ source code archive and associated metadata.
+ The deposit is directly marked with status ``deposited``.
+
+* a multisteps deposit:
+
+ 1. Create an incomplete deposit (marked with status ``partial``)
+ 2. Add data to a deposit (in multiple requests if needed)
+ 3. Finalize deposit (the status becomes ``deposited``)
+
+
+Overall, a deposit can be a in series of steps as follow:
+
+.. figure:: images/status.svg
+ :alt:
+
+The important things to notice for now is that it can be:
+
+:partial: the deposit is new or partially received since it
+ can be done in multiple requests
+:expired: deposit has been there too long and is now deemed
+ ready to be garbage collected
+:deposited: deposit complete, it is ready to be checked to ensure data consistency
+:verified: deposit is fully received, checked, and ready for loading
+:loading: loading is ongoing on swh's side
+:done: loading is successful
+:failed: loading is a failure
+
+
+When you push a deposit, it is either in the `deposited` state or in the
+`partial` state if you asked for a partial upload.
+
+
+
+Single deposit
+^^^^^^^^^^^^^^
+
+Once the files are ready for deposit, we want to do the actual deposit in one
+shot, i.e. sending both the archive (zip) file and the metadata file.
+
+* 1 archive (content-type ``application/zip`` or ``application/x-tar``)
+* 1 metadata file in atom xml format (``content-type: application/atom+xml;type=entry``)
+
+For this, we need to provide the:
+
+* arguments: ``--username 'name' --password 'pass'`` as credentials
+* archive's path (example: ``--archive path/to/archive-name.tgz``)
+* metadata file path (example: ``--metadata path/to/metadata.xml``)
+
+to the `swh deposit upload` command.
+
+
+
+Example:
+
+To push the Belenios 1.12 we prepared previously on the testing instance of the
+deposit:
+
+.. code:: console
+
+ (deposit)$ ls
+ belenios-1.12.zip metadata.xml deposit
+ (deposit)$ swh deposit upload --username <name> --password <secret> \
+ --url https://deposit.staging.swh.network/1 \
+ --slug belenios-01243065 \
+ --archive belenios.zip \
+ --metadata metadata.xml \
+ --format json | jq
+ {
+ 'deposit_status': 'deposited',
+ 'deposit_id': '1',
+ 'deposit_date': 'Oct. 28, 2020, 1:52 p.m.',
+ 'deposit_status_detail': None
+ }
+
+ (deposit)$
+
+
+You just posted a deposit to your main collection on Software Heritage (staging
+area)!
+
+The returned value is a JSON dict, in which you will notably find the deposit
+id (needed to check for its status later on) and the current status, which
+should be `deposited` if no error has occurred.
+
+Note: As the deposit is in ``deposited`` status, you can no longer
+update the deposit after this query. It will be answered with a 403
+(Forbidden) answer.
+
+If something went wrong, an equivalent response will be given with the
+`error` and `detail` keys explaining the issue, e.g.:
+
+.. code:: console
+
+ {
+ 'error': 'Unknown collection name xyz',
+ 'detail': None,
+ 'deposit_status': None,
+ 'deposit_status_detail': None,
+ 'deposit_swh_id': None,
+ 'status': 404
+ }
+
+
+Once the deposit has been done, you can check its status using the `swh deposit
+status` command:
+
+.. code:: console
+
+ (deposit)$ swh deposit status --username <name> --password <secret> \
+ --url https://deposit.staging.swh.network/1 \
+ --deposit-id 1 -f json | jq
+ {
+ "deposit_id": "1",
+ "deposit_status": "done",
+ "deposit_status_detail": "The deposit has been successfully loaded into the Software Heritage archive",
+ "deposit_swh_id": "swh:1:dir:63a6fc0ed8f69bf66ccbf99fc0472e30ef0a895a",
+ "deposit_swh_id_context": "swh:1:dir:63a6fc0ed8f69bf66ccbf99fc0472e30ef0a895a;origin=https://softwareheritage.org/belenios-01234065;visit=swh:1:snp:0ae536667689da7047bfb7aa9f37f5958e9f4647;anchor=swh:1:rev:17ad98c940104d45b6b6bd6fba9aa832eeb95638;path=/",
+ "deposit_external_id": "belenios-01234065"
+ }
+
+
+Single deposit with a metadata file
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Same principle as above, but now we will provide a metadata file with the
+artifact to be archived. When using this method, there is need for giving extra
+options like `--name` etc.
+
+.. code:: console
+
+ (deposit)$ swh deposit upload --username <name> --password <secret> \
+ --url https://deposit.staging.swh.network/1 \
+ --format json \
+ --metadata metadata.xml \
+ --archive belenios.zip | jq
+ {
+ "deposit_status": "deposited",
+ "deposit_id": "3",
+ "deposit_date": "Oct. 28, 2020, 1:59 p.m.",
+ "deposit_status_detail": None
+ }
+
+
+Multisteps deposit
+^^^^^^^^^^^^^^^^^^
+The steps to create a multisteps deposit:
+
+1. Create an incomplete deposit
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+First use the ``--partial`` argument to declare there is more to come
+
+.. code:: console
+
+ $ swh deposit upload --username name --password secret \
+ --archive foo.tar.gz \
+ --partial
+
+
+2. Add content or metadata to the deposit
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Continue the deposit by using the ``--deposit-id`` argument given as a response
+for the first step. You can continue adding content or metadata while you use
+the ``--partial`` argument.
+
+To only add one new archive to the deposit:
+
+.. code:: console
+
+ $ swh deposit upload --username name --password secret \
+ --archive add-foo.tar.gz \
+ --deposit-id 42 \
+ --partial
+
+To only add metadata to the deposit:
+
+.. code:: console
+
+ $ swh deposit upload --username name --password secret \
+ --metadata add-foo.tar.gz.metadata.xml \
+ --deposit-id 42 \
+ --partial
+
+or:
+
+.. code:: console
+
+ $ swh deposit upload --username name --password secret \
+ --name 'add-foo' --author 'someone' \
+ --deposit-id 42 \
+ --partial
+
+
+3. Finalize deposit
+~~~~~~~~~~~~~~~~~~~
+
+On your last addition (same command as before), by not declaring it
+``--partial``, the deposit will be considered completed. Its status will be
+changed to ``deposited``
+
+
+Update deposit
+----------------
+* replace deposit:
+
+ - only possible if the deposit status is ``partial`` and
+ ``--deposit-id <id>`` is provided
+
+ - by using the ``--replace`` flag
+
+ - ``--metadata-deposit`` replaces associated existing metadata
+ - ``--archive-deposit`` replaces associated archive(s)
+ - by default, with no flag or both, you'll replace associated
+ metadata and archive(s):
+
+.. code:: console
+
+ $ swh deposit upload --username name --password secret \
+ --deposit-id 11 \
+ --archive updated-je-suis-gpl.tgz \
+ --replace
+
+* update a loaded deposit with a new version:
+
+ - by using the external-id with the ``--slug`` argument, you will
+ link the new deposit with its parent deposit:
+
+.. code:: console
+
+ $ swh deposit upload --username name --password secret \
+ --archive je-suis-gpl-v2.tgz \
+ --slug 'je-suis-gpl' \
+
+
+
+Check the deposit's status
+--------------------------
+
+You can check the status of the deposit by using the ``--deposit-id`` argument:
+
+.. code:: console
+
+ $ swh deposit status --username name --password secret \
+ --deposit-id 11
+
+.. code:: json
+
+ {
+ "deposit_id": 11,
+ "deposit_status": "deposited",
+ "deposit_swh_id": null,
+ "deposit_status_detail": "Deposit is ready for additional checks \
+ (tarball ok, metadata, etc...)"
+ }
+
+The different statuses:
+
+- **partial**: multipart deposit is still ongoing
+- **deposited**: deposit completed
+- **rejected**: deposit failed the checks
+- **verified**: content and metadata verified
+- **loading**: loading in-progress
+- **done**: loading completed successfully
+- **failed**: the deposit loading has failed
+
+When the deposit has been loaded into the archive, the status will be
+marked ``done``. In the response, will also be available the
+<deposit_swh_id>, <deposit_swh_id_context>. For example:
+
+.. code:: json
+
+ {
+ "deposit_id": 11,
+ "deposit_status": "done",
+ "deposit_swh_id": "swh:1:dir:d83b7dda887dc790f7207608474650d4344b8df9",
+ "deposit_swh_id_context": "swh:1:dir:d83b7dda887dc790f7207608474650d4344b8df9;origin=https://forge.softwareheritage.org/source/jesuisgpl/;visit=swh:1:snp:68c0d26104d47e278dd6be07ed61fafb561d0d20;anchor=swh:1:rev:e76ea49c9ffbb7f73611087ba6e999b19e5d71eb;path=/",
+ "deposit_status_detail": "The deposit has been successfully \
+ loaded into the Software Heritage archive"
+ }
File Metadata
Details
Attached
Mime Type
text/plain
Expires
Nov 5 2024, 7:17 AM (8 w, 2 d ago)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
3230075
Attached To
D4430: doc: rename Getting Started as User Manual and update the content
Event Timeline
Log In to Comment