Changeset View
Changeset View
Standalone View
Standalone View
docs/api.rst
Show All 19 Lines | |||||
Unless otherwise stated, API endpoints respond to HTTP GET method. | Unless otherwise stated, API endpoints respond to HTTP GET method. | ||||
Object identification | Object identification | ||||
--------------------- | --------------------- | ||||
The vault stores bundles corresponding to different kinds of objects (see | The vault stores bundles corresponding to different kinds of objects (see | ||||
:ref:`data-model`). The following object kinds are currently supported by the | :ref:`data-model`). | ||||
Vault: | |||||
- directories | The URL fragment ``:bundletype/:swhid`` is used throughout the vault API to | ||||
- revisions | identify vault objects. See :ref:`persistent-identifiers` for details on | ||||
- snapshots | the syntax and meaning of ``:swhid``. | ||||
The URL fragment ``:objectkind/:objectid`` is used throughout the vault API to | |||||
identify vault objects. The syntax and meaning of ``:objectid`` for the | |||||
different object kinds is detailed below. | |||||
In the case of revisions, a third parameter, ``:format``, must be used to | Bundle types | ||||
specify the format of the resulting bundle. The URL fragment then becomes | ------------ | ||||
``:objectkind/:objectid/:format``. | |||||
Directories | Flat | ||||
~~~~~~~~~~~ | ~~~~ | ||||
- object kind: ``directory`` | Flat bundles are simple tarballs that can be read without any specialized software. | ||||
- URL fragment: ``directory/:dir_id`` | |||||
where ``:dir_id`` is a :py:func:`directory identifier | When cooking directories, they are (very close to) the original directories that | ||||
<swh.model.identifiers.directory_identifier>`. | were ingested. | ||||
When cooking other types of objects, they have multiple root directories, | |||||
each corresponding to an original object (revision, ...) | |||||
The only format available for a directory export is a gzip-compressed | This is typically only useful to cook directories; cooking other types of objects | ||||
tarball. You can extract the resulting bundle using: | (revisions, releases, snapshots) are usually done with ``git-bare`` as it is | ||||
more efficient and closer to the original repository. | |||||
You can extract the resulting bundle using: | |||||
.. code:: shell | .. code:: shell | ||||
tar xaf bundle.tar.gz | tar xaf bundle.tar.gz | ||||
Revisions | gitfast | ||||
~~~~~~~~~ | ~~~~~~~ | ||||
- object kind: ``revision`` | |||||
- URL fragment: ``revision/:rev_id/:format`` | |||||
where ``:rev_id`` is a :py:func:`revision identifier | |||||
<swh.model.identifiers.revision_identifier>` and ``:format`` is the export | |||||
format. | |||||
The only format available for a revision export is ``gitfast``: a | A gzip-compressed `git fast-export | ||||
gzip-compressed `git fast-export | |||||
<https://git-scm.com/docs/git-fast-export>`_. You can extract the resulting | <https://git-scm.com/docs/git-fast-export>`_. You can extract the resulting | ||||
bundle using: | bundle using: | ||||
.. code:: shell | .. code:: shell | ||||
git init | git init | ||||
zcat bundle.gitfast.gz | git fast-import | zcat bundle.gitfast.gz | git fast-import | ||||
git checkout HEAD | git checkout HEAD | ||||
Repository snapshots | git-bare | ||||
~~~~~~~~~~~~~~~~~~~~ | ~~~~~~~~ | ||||
A tarball that can be decompressed to get a real git repository. | |||||
It is without a checkout, so it is the equivalent of what one would get | |||||
with ``git clone --bare``. | |||||
This is the most flexible bundle type, as it allow to perfectly recreate | |||||
original git repositories, including branches. | |||||
You can extract the resulting bundle using: | |||||
.. TODO | .. code:: shell | ||||
tar xaf bundle.tar.gz | |||||
**(NOT AVAILABLE YET)** | Then explore its content like a normal ("non-bare") git repository by cloning it: | ||||
- object kind: ``snapshot`` | .. code:: shell | ||||
- URL fragment: ``snapshot/:snp_id`` | |||||
where ``:snp_id`` is a :py:func:`snapshot identifier | git clone path/to/extracted/:swhid | ||||
<swh.model.identifiers.snapshot_identifier>`. | |||||
Cooking and status checking | Cooking and status checking | ||||
--------------------------- | --------------------------- | ||||
Vault bundles might be ready for retrieval or not. When they are not, they will | Vault bundles might be ready for retrieval or not. When they are not, they will | ||||
need to be **cooked** before they can be retrieved. A cooked bundle will remain | need to be **cooked** before they can be retrieved. A cooked bundle will remain | ||||
around until it expires; after expiration, it will need to be cooked again | around until it expires; after expiration, it will need to be cooked again | ||||
before it can be retrieved. Cooking is idempotent, and a no-op in between a | before it can be retrieved. Cooking is idempotent, and a no-op in between a | ||||
previous cooking operation and expiration. | previous cooking operation and expiration. | ||||
.. http:post:: /vault/:objectkind/:objectid[/:format] | .. http:post:: /vault/:bundletype/:swhid | ||||
.. http:get:: /vault/:objectkind/:objectid[/:format] | .. http:get:: /vault/:bundletype/:swhid | ||||
**Request body**: optionally, an ``email`` POST parameter containing an | **Request body**: optionally, an ``email`` POST parameter containing an | ||||
e-mail to notify when the bundle cooking has ended. | e-mail to notify when the bundle cooking has ended. | ||||
**Allowed HTTP Methods:** | **Allowed HTTP Methods:** | ||||
- :http:method:`post` to **request** a bundle cooking | - :http:method:`post` to **request** a bundle cooking | ||||
- :http:method:`get` to check the progress and status of the cooking | - :http:method:`get` to check the progress and status of the cooking | ||||
- :http:method:`head` | - :http:method:`head` | ||||
- :http:method:`options` | - :http:method:`options` | ||||
**Response:** | **Response:** | ||||
:statuscode 200: bundle available for cooking, status of the cooking | :statuscode 200: bundle available for cooking, status of the cooking | ||||
:statuscode 400: malformed identifier hash or format | :statuscode 400: malformed SWHID | ||||
:statuscode 404: unavailable bundle or object not found | :statuscode 404: unavailable bundle or object not found | ||||
.. sourcecode:: http | .. sourcecode:: http | ||||
HTTP/1.1 200 OK | HTTP/1.1 200 OK | ||||
Content-Type: application/json | Content-Type: application/json | ||||
{ | { | ||||
"id": 42, | "id": 42, | ||||
"fetch_url": "/api/1/vault/directory/:dir_id/raw/", | "fetch_url": "/api/1/vault/flat/:swhid/raw/", | ||||
"obj_id": ":dir_id", | "swhid": ":swhid", | ||||
"obj_type": "directory", | |||||
"progress_message": "Creating tarball...", | "progress_message": "Creating tarball...", | ||||
"status": "pending" | "status": "pending" | ||||
} | } | ||||
After a cooking request has been started, all subsequent GET and POST | After a cooking request has been started, all subsequent GET and POST | ||||
requests to the cooking URL return some JSON data containing information | requests to the cooking URL return some JSON data containing information | ||||
about the progress of the bundle creation. The JSON contains the | about the progress of the bundle creation. The JSON contains the | ||||
following keys: | following keys: | ||||
- ``id``: the ID of the cooking request | - ``id``: the ID of the cooking request | ||||
- ``fetch_url``: the URL that can be used for the retrieval of the bundle | - ``fetch_url``: the URL that can be used for the retrieval of the bundle | ||||
- ``obj_type``: an internal identifier uniquely representing the object | - ``swhid``: the identifier of the requested bundle | ||||
kind and the format of the required bundle. | |||||
- ``obj_id``: the identifier of the requested bundle | |||||
- ``progress_message``: a string describing the current progress of the | - ``progress_message``: a string describing the current progress of the | ||||
cooking. If the cooking failed, ``progress_message`` will contain the | cooking. If the cooking failed, ``progress_message`` will contain the | ||||
reason of the failure. | reason of the failure. | ||||
- ``status``: one of the following values: | - ``status``: one of the following values: | ||||
- ``new``: the bundle request was created | - ``new``: the bundle request was created | ||||
- ``pending``: the bundle is being cooked | - ``pending``: the bundle is being cooked | ||||
- ``done``: the bundle has been cooked and is ready for retrieval | - ``done``: the bundle has been cooked and is ready for retrieval | ||||
- ``failed``: the bundle cooking failed and can be retried | - ``failed``: the bundle cooking failed and can be retried | ||||
Retrieval | Retrieval | ||||
--------- | --------- | ||||
Retrieve a specific bundle from the vault with: | Retrieve a specific bundle from the vault with: | ||||
.. http:get:: /vault/:objectkind/:objectid[/:format]/raw | .. http:get:: /vault/:bundletype/:swhid/raw | ||||
Where ``:format`` is optional, depending on the object kind. | |||||
**Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, | **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, | ||||
:http:method:`options` | :http:method:`options` | ||||
**Response**: | **Response**: | ||||
:statuscode 200: bundle available; response body is the bundle. | :statuscode 200: bundle available; response body is the bundle. | ||||
:statuscode 404: unavailable bundle; client should request its cooking. | :statuscode 404: unavailable bundle; client should request its cooking. |