diff --git a/docs/getting-started.rst b/docs/getting-started.rst index ff561bf0..6e281ff8 100644 --- a/docs/getting-started.rst +++ b/docs/getting-started.rst @@ -1,321 +1,320 @@ Getting Started =============== This is a guide for how to prepare and push a software deposit with the `swh deposit` commands. The API is rooted at https://deposit.softwareheritage.org/1. For more details, see the `main documentation <./index.html>`__. Requirements ------------ You need to be referenced on SWH's client list to have: * credentials (needed for the basic authentication step) - in this document we reference ```` as the client's name and ```` as its associated authentication password. * an associated collection_. .. _collection: https://bitworking.org/projects/atom/rfc5023#rfc.section.8.3.3 `Contact us for more information. `__ Prepare a deposit ----------------- * compress the files in a supported archive format: - zip: common zip archive (no multi-disk zip files). - tar: tar archive without compression or optionally any of the following compression algorithm gzip (`.tar.gz`, `.tgz`), bzip2 (`.tar.bz2`) , or lzma (`.tar.lzma`) * prepare a metadata file (`more details <./metadata.html>`__.): - specify metadata schema/vocabulary (CodeMeta is strongly recommended) - specify *MUST* metadata (url, authors, software name and the external\_identifier) - add all available information under the compatible metadata term. Here is an example of an atom entry file with CodeMeta terms: .. code:: xml Je suis GPL swh je-suis-gpl https://forge.softwareheritage.org/source/jesuisgpl/ 2018-01-05 Je suis GPL is a modified version of GNU Hello whose sole purpose is to showcase the usage of Software Heritage for license compliance purposes. 0.1 GNU/Linux stable C GNU General Public License v3.0 or later https://spdx.org/licenses/GPL-3.0-or-later.html Stefano Zacchiroli Maintainer Push deposit ------------ You can push a deposit with: * a single deposit (archive + metadata): The user posts in one query a software source code archive and associated metadata. The deposit is directly marked with status ``deposited``. * a multisteps deposit: 1. Create an incomplete deposit (marked with status ``partial``) 2. Add data to a deposit (in multiple requests if needed) 3. Finalize deposit (the status becomes ``deposited``) Single deposit ^^^^^^^^^^^^^^ Once the files are ready for deposit, we want to do the actual deposit in one shot, sending exactly one POST query: * 1 archive (content-type ``application/zip`` or ``application/x-tar``) * 1 metadata file in atom xml format (``content-type: application/atom+xml;type=entry``) For this, we need to provide the: * arguments: ``--username 'name' --password 'pass'`` as credentials * archive's path (example: ``--archive path/to/archive-name.tgz``) : * (optionally) metadata file's path ``--metadata path/to/file.metadata.xml``. If not provided, the archive's filename will be used to determine the metadata file, e.g: ``path/to/archive-name.tgz.metadata.xml`` * (optionally) ``--slug 'your-id'`` argument, a reference to a unique identifier the client uses for the software object. You can do this with the following command: minimal deposit .. code:: shell $ swh deposit upload --username name --password secret \ --archive je-suis-gpl.tgz with client's external identifier (``slug``) .. code:: shell $ swh deposit upload --username name --password secret \ --archive je-suis-gpl.tgz \ --slug je-suis-gpl to a specific client's collection .. code:: shell $ swh deposit upload --username name --password secret \ --archive je-suis-gpl.tgz \ --collection 'second-collection' You just posted a deposit to your collection on Software Heritage If everything went well, the successful response will contain the elements below: .. code:: shell { 'deposit_status': 'deposited', 'deposit_id': '7', 'deposit_date': 'Jan. 29, 2018, 12:29 p.m.' } Note: As the deposit is in ``deposited`` status, you can no longer update the deposit after this query. It will be answered with a 403 forbidden answer. If something went wrong, an equivalent response will be given with the `error` and `detail` keys explaining the issue, e.g.: .. code:: shell { 'error': 'Unknown collection name xyz', 'detail': None, 'deposit_status': None, 'deposit_status_detail': None, 'deposit_swh_id': None, 'status': 404 } multisteps deposit ^^^^^^^^^^^^^^^^^^^^^^^^^ The steps to create a multisteps deposit: 1. Create an incomplete deposit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ First use the ``--partial`` argument to declare there is more to come .. code:: shell $ swh deposit upload --username name --password secret \ --archive foo.tar.gz \ --partial 2. Add content or metadata to the deposit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Continue the deposit by using the ``--deposit-id`` argument given as a response for the first step. You can continue adding content or metadata while you use the ``--partial`` argument. .. code:: shell $ swh deposit upload --username name --password secret \ --archive add-foo.tar.gz \ --deposit-id 42 \ --partial In case you want to add only one new archive without metadata: .. code:: shell $ swh deposit upload --username name --password secret \ --archive add-foo.tar.gz \ --archive-deposit \ --deposit-id 42 \ --partial If you want to add only metadata, use: .. code:: shell $ swh deposit upload --username name --password secret \ --metadata add-foo.tar.gz.metadata.xml \ --metadata-deposit \ --deposit-id 42 \ --partial 3. Finalize deposit ~~~~~~~~~~~~~~~~~~~ On your last addition, by not declaring it as ``--partial``, the deposit will be considered as completed and its status will be changed to ``deposited``: .. code:: shell $ swh deposit upload --username name --password secret \ --metadata add-foo.tar.gz.metadata.xml \ --metadata-deposit \ --deposit-id 42 Update deposit ---------------- * replace deposit: - only possible if the deposit status is ``partial`` and ``--deposit-id `` is provided - by using the ``--replace`` flag - ``--metadata-deposit`` replaces associated existing metadata - ``--archive-deposit`` replaces associated archive(s) - by default, with no flag or both, you'll replace associated metadata and archive(s): .. code:: shell $ swh deposit upload --username name --password secret \ --deposit-id 11 \ --archive updated-je-suis-gpl.tgz \ --replace * update a loaded deposit with a new version: - by using the external-id with the ``--slug`` argument, you will link the new deposit with its parent deposit: .. code:: shell $ swh deposit upload --username name --password secret \ --archive je-suis-gpl-v2.tgz \ --slug 'je-suis-gpl' \ Check the deposit's status -------------------------- You can check the status of the deposit by using the ``--deposit-id`` argument: .. code:: shell - $ swh deposit upload --username name --password secret \ - --deposit-id 11 \ - --status + $ swh deposit status --username name --password secret \ + --deposit-id 11 .. code:: json { 'deposit_id': '11', 'deposit_status': 'deposited', 'deposit_swh_id': None, 'deposit_status_detail': 'Deposit is ready for additional checks \ (tarball ok, metadata, etc...)' } The different statuses: - **partial**: multipart deposit is still ongoing - **deposited**: deposit completed - **rejected**: deposit failed the checks - **verified**: content and metadata verified - **loading**: loading in-progress - **done**: loading completed successfully - **failed**: the deposit loading has failed When the deposit has been loaded into the archive, the status will be marked ``done``. In the response, will also be available the , , , . For example: .. code:: json { 'deposit_id': '11', 'deposit_status': 'done', 'deposit_swh_id': 'swh:1:dir:d83b7dda887dc790f7207608474650d4344b8df9', 'deposit_swh_id_context': 'swh:1:dir:d83b7dda887dc790f7207608474650d4344b8df9;origin=https://forge.softwareheritage.org/source/jesuisgpl/', 'deposit_swh_anchor_id': 'swh:1:rev:e76ea49c9ffbb7f73611087ba6e999b19e5d71eb', 'deposit_swh_anchor_id_context': 'swh:1:rev:e76ea49c9ffbb7f73611087ba6e999b19e5d71eb;origin=https://forge.softwareheritage.org/source/jesuisgpl/', 'deposit_status_detail': 'The deposit has been successfully \ loaded into the Software Heritage archive' } diff --git a/swh/deposit/cli/client.py b/swh/deposit/cli/client.py index bb06d26a..43eed610 100644 --- a/swh/deposit/cli/client.py +++ b/swh/deposit/cli/client.py @@ -1,353 +1,385 @@ # Copyright (C) 2017-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information import os import logging import tempfile import uuid import click import xmltodict from swh.deposit.client import PublicApiDepositClient from swh.deposit.cli import deposit logger = logging.getLogger(__name__) class InputError(ValueError): """Input script error """ pass def generate_slug(): """Generate a slug (sample purposes). """ return str(uuid.uuid4()) def generate_metadata_file(name, external_id, authors): """Generate a temporary metadata file with the minimum required metadata This generates a xml file in a temporary location and returns the path to that file. This is up to the client of that function to clean up the temporary file. Args: name (str): Software's name external_id (str): External identifier (slug) or generated one authors (List[str]): List of author names Returns: Filepath to the metadata generated file """ _, tmpfile = tempfile.mkstemp(prefix='swh.deposit.cli.') # generate a metadata file with the minimum required metadata codemetadata = { 'entry': { '@xmlns': "http://www.w3.org/2005/Atom", '@xmlns:codemeta': "https://doi.org/10.5063/SCHEMA/CODEMETA-2.0", 'codemeta:name': name, 'codemeta:identifier': external_id, 'codemeta:author': [{ 'codemeta:name': author_name } for author_name in authors], }, } logging.debug('Temporary file: %s', tmpfile) logging.debug('Metadata dict to generate as xml: %s', codemetadata) s = xmltodict.unparse(codemetadata, pretty=True) logging.debug('Metadata dict as xml generated: %s', s) with open(tmpfile, 'w') as fp: fp.write(s) return tmpfile def _cleanup_tempfile(config): """Clean up the temporary metadata file generated. Args: config (Dict): A configuration dict with 2 important keys for that routine, 'cleanup_tempfile' (bool) and 'metadata' (path to eventually clean up) """ if config['cleanup_tempfile']: path = config['metadata'] if os.path.exists(path): os.unlink(path) +def _client(url, username, password): + """Instantiate a client to access the deposit api server + + Args: + url (str): Deposit api server + username (str): User + password (str): User's password + + """ + client = PublicApiDepositClient({ + 'url': url, + 'auth': { + 'username': username, + 'password': password + }, + }) + return client + + +def _collection(client): + """Retrieve the client's collection + + """ + # retrieve user's collection + sd_content = client.service_document() + if 'error' in sd_content: + raise InputError('Service document retrieval: %s' % ( + sd_content['error'], )) + collection = sd_content[ + 'service']['workspace']['collection']['sword:name'] + return collection + + def client_command_parse_input( username, password, archive, metadata, archive_deposit, metadata_deposit, collection, slug, partial, deposit_id, replace, - url, status, name, authors): + url, name, authors): """Parse the client subcommand options and make sure the combination is acceptable*. If not, an InputError exception is raised explaining the issue. By acceptable, we mean: - A multipart deposit (create or update) needs both an existing software archive and an existing metadata file - A binary deposit (create/update) needs an existing software archive - A metadata deposit (create/update) needs an existing metadata file - A deposit update needs a deposit_id to be provided This won't prevent all failure cases though. The remaining errors are already dealt with the underlying api client. Raises: InputError explaining the issue Returns: dict with the following keys: 'archive': the software archive to deposit 'username': username 'password': associated password 'metadata': the metadata file to deposit 'collection': the username's associated client 'slug': the slug or external id identifying the deposit to make 'partial': if the deposit is partial or not 'client': instantiated class 'url': deposit's server main entry point 'deposit_type': deposit's type (binary, multipart, metadata) 'deposit_id': optional deposit identifier """ cleanup_tempfile = False try: - if status and not deposit_id: - raise InputError("Deposit id must be provided for status check") - - if status and deposit_id: # status is higher priority over deposit - archive_deposit = False - metadata_deposit = False - archive = None - metadata = None - if archive_deposit and metadata_deposit: # too many flags use, remove redundant ones (-> multipart deposit) archive_deposit = False metadata_deposit = False if archive and not os.path.exists(archive): raise InputError('Software Archive %s must exist!' % archive) if not slug: # generate one as this is mandatory slug = generate_slug() if archive and not metadata: # we need to have the metadata if name and authors: metadata = generate_metadata_file(name, slug, authors) cleanup_tempfile = True else: raise InputError('Either metadata deposit file or (`--name` ' ' and `--author`) fields must be provided') if metadata_deposit: archive = None if archive_deposit: metadata = None if metadata_deposit and not metadata: raise InputError( "Metadata deposit filepath must be provided for metadata " "deposit") if metadata and not os.path.exists(metadata): raise InputError('Software Archive metadata %s must exist!' % ( metadata, )) - if not status and not archive and not metadata: + if not archive and not metadata: raise InputError( 'Please provide an actionable command. See --help for more ' 'information.') if replace and not deposit_id: raise InputError( 'To update an existing deposit, you must provide its id') - client = PublicApiDepositClient({ - 'url': url, - 'auth': { - 'username': username, - 'password': password - }, - }) + client = _client(url, username, password) if not collection: - # retrieve user's collection - sd_content = client.service_document() - if 'error' in sd_content: - raise InputError('Service document retrieval: %s' % ( - sd_content['error'], )) - collection = sd_content[ - 'service']['workspace']['collection']['sword:name'] + collection = _collection(client) return { 'archive': archive, 'username': username, 'password': password, 'metadata': metadata, 'cleanup_tempfile': cleanup_tempfile, 'collection': collection, 'slug': slug, 'in_progress': partial, 'client': client, 'url': url, 'deposit_id': deposit_id, 'replace': replace, } except Exception: # to be clean, cleanup prior to raise _cleanup_tempfile({ 'cleanup_tempfile': cleanup_tempfile, 'metadata': metadata }) raise def _subdict(d, keys): 'return a dict from d with only given keys' return {k: v for k, v in d.items() if k in keys} -def deposit_status(config, logger): - logger.debug('Status deposit') - keys = ('collection', 'deposit_id') - client = config['client'] - return client.deposit_status( - **_subdict(config, keys)) - - def deposit_create(config, logger): """Delegate the actual deposit to the deposit client. """ logger.debug('Create deposit') client = config['client'] keys = ('collection', 'archive', 'metadata', 'slug', 'in_progress') return client.deposit_create( **_subdict(config, keys)) def deposit_update(config, logger): """Delegate the actual deposit to the deposit client. """ logger.debug('Update deposit') client = config['client'] keys = ('collection', 'deposit_id', 'archive', 'metadata', 'slug', 'in_progress', 'replace') return client.deposit_update( **_subdict(config, keys)) @deposit.command() @click.option('--username', required=1, help="(Mandatory) User's name") @click.option('--password', required=1, help="(Mandatory) User's associated password") @click.option('--archive', help='(Optional) Software archive to deposit') @click.option('--metadata', help="(Optional) Path to xml metadata file. If not provided, this will use a file named .metadata.xml") # noqa @click.option('--archive-deposit/--no-archive-deposit', default=False, help='(Optional) Software archive only deposit') @click.option('--metadata-deposit/--no-metadata-deposit', default=False, help='(Optional) Metadata only deposit') @click.option('--collection', help="(Optional) User's collection. If not provided, this will be fetched.") # noqa @click.option('--slug', help="""(Optional) External system information identifier. If not provided, it will be generated""") # noqa @click.option('--partial/--no-partial', default=False, help='(Optional) The deposit will be partial, other deposits will have to take place to finalize it.') # noqa @click.option('--deposit-id', default=None, help='(Optional) Update an existing partial deposit with its identifier') # noqa @click.option('--replace/--no-replace', default=False, help='(Optional) Update by replacing existing metadata to a deposit') # noqa @click.option('--url', default='https://deposit.softwareheritage.org/1', help="(Optional) Deposit server api endpoint. By default, https://deposit.softwareheritage.org/1") # noqa -@click.option('--status/--no-status', default=False, - help="(Optional) Deposit's status") @click.option('--verbose/--no-verbose', default=False, help='Verbose mode') @click.option('--name', help='Software name') @click.option('--author', multiple=True, help='Software author(s), this can be repeated as many times' ' as there are authors') @click.pass_context def upload(ctx, username, password, archive=None, metadata=None, archive_deposit=False, metadata_deposit=False, collection=None, slug=None, partial=False, deposit_id=None, - replace=False, status=False, + replace=False, url='https://deposit.softwareheritage.org/1', verbose=False, name=None, author=None): """Software Heritage Public Deposit Client - Create/Update deposit through the command line or access its - status. + Create/Update deposit through the command line. More documentation can be found at https://docs.softwareheritage.org/devel/swh-deposit/getting-started.html. """ config = {} try: logger.debug('Parsing cli options') config = client_command_parse_input( username, password, archive, metadata, archive_deposit, metadata_deposit, collection, slug, partial, deposit_id, - replace, url, status, name, author) + replace, url, name, author) except InputError as e: msg = 'Problem during parsing options: %s' % e r = { 'error': msg, } logger.info(r) return 1 try: if verbose: logger.info("Parsed configuration: %s" % ( config, )) deposit_id = config['deposit_id'] - if status and deposit_id: - r = deposit_status(config, logger) - elif not status and deposit_id: + if deposit_id: r = deposit_update(config, logger) - elif not status and not deposit_id: + else: r = deposit_create(config, logger) logger.info(r) finally: _cleanup_tempfile(config) + + +@deposit.command() +@click.option('--url', default='https://deposit.softwareheritage.org/1', + help="(Optional) Deposit server api endpoint. By default, " + "https://deposit.softwareheritage.org/1") +@click.option('--username', required=1, + help="(Mandatory) User's name") +@click.option('--password', required=1, + help="(Mandatory) User's associated password") +@click.option('--deposit-id', default=None, + required=1, + help="Deposit identifier.") +@click.pass_context +def status(ctx, url, username, password, deposit_id): + """Deposit's status + + """ + logger.debug('Status deposit') + try: + client = _client(url, username, password) + collection = _collection(client) + except InputError as e: + msg = 'Problem during parsing options: %s' % e + r = { + 'error': msg, + } + logger.info(r) + return 1 + + r = client.deposit_status( + collection=collection, deposit_id=deposit_id) + logger.info(r)