Page MenuHomeSoftware Heritage
Files F9341052

No OneTemporary
Actions

View Options

diff --git a/PKG-INFO b/PKG-INFO
index 12406ca9..7c7e304d 100644
--- a/PKG-INFO
+++ b/PKG-INFO
@@ -1,98 +1,98 @@
Metadata-Version: 2.1
Name: swh.deposit
-Version: 0.14.2
+Version: 0.14.3
Summary: Software Heritage Deposit Server
Home-page: https://forge.softwareheritage.org/source/swh-deposit/
Author: Software Heritage developers
Author-email: swh-devel@inria.fr
License: UNKNOWN
Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest
Project-URL: Funding, https://www.softwareheritage.org/donate
Project-URL: Source, https://forge.softwareheritage.org/source/swh-deposit
Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-deposit/
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 5 - Production/Stable
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Provides-Extra: testing
Provides-Extra: server
License-File: LICENSE
License-File: AUTHORS
Software Heritage - Deposit
===========================
Simple Web-Service Offering Repository Deposit (S.W.O.R.D) is an interoperability
standard for digital file deposit.
This repository is both the `SWORD v2`_ Server and a deposit command-line client
implementations.
This implementation allows interaction between a client (a repository) and a server (SWH
repository) to deposit software source code archives and associated metadata.
Description
-----------
Most of the software source code artifacts present in the SWH Archive are gathered by
the mean of :term:`loader <loader>` workers run by the SWH project from sourve code
origins identified by :term:`lister <lister>` workers. This is a pull mechanism: it's
the responsibility of the SWH project to gather and collect source code artifacts that
way.
Alternatively, SWH allows its partners to push source code artifacts and metadata
directly into the Archive with a push-based mechanism. By using this possibility
different actors, holding software artifacts or metadata, can preserve their assets
without having to pass through an intermediate collaborative development platform, which
is already harvested by SWH (e.g GitHub, Gitlab, etc.).
This mechanism is the `deposit`.
The main idea is the deposit is an authenticated access to an API allowing the user to
provide source code artifacts -- with metadata -- to be ingested in the SWH Archive. The
result of that is a :ref:`SWHID <persistent-identifiers>` that can be used to uniquely
and persistently identify that very piece of source code.
This unique identifier can then be used to `reference the source code
<https://hal.archives-ouvertes.fr/hal-02446202>`_ (e.g. in a `scientific paper
<https://www.softwareheritage.org/2020/05/26/citing-software-with-style/>`_) and
retrieve it using the :ref:`vault <swh-vault>` feature of the SWH Archive platform.
The differences between a piece of code uploaded using the deposit rather than simply
asking SWH to archive a repository using the `save code now
<https://archive.softwareheritage.org/save/>`_ feature are:
- a deposited artifact is provided from one of the SWH partners which is regarded as a
trusted authority,
- a deposited artifact requires metadata properties describing the source code artifact,
- a deposited artifact has a codemeta_ metadata entry attached to it,
- a deposited artifact has the same visibility on the SWH Archive than a collected
repository,
- a deposited artifact can be searched with its provided url property on the SWH
Archive,
- the deposit API uses the `SWORD v2`_ API, thus requires some tooling to send deposits
to SWH. These tools are provided with this repository.
See the :ref:`deposit-user-manual` page for more details on how to use the deposit client
command line tools to push a deposit in the SWH Archive.
See the :ref:`deposit-api-specifications` reference pages of the SWORDv2 API implementation
in `swh.deposit` if you want to do upload deposits using HTTP requests.
Read the :ref:`deposit-metadata` chapter to get more details on what metadata
are supported when doing a deposit.
See :ref:`swh-deposit-dev-env` if you want to hack the code of the `swh.deposit` module.
See :ref:`swh-deposit-prod-env` if you want to deploy your own copy of the
`swh.deposit` stack.
.. _codemeta: https://codemeta.github.io/
.. _`SWORD v2`: http://swordapp.org/sword-v2/
diff --git a/debian/changelog b/debian/changelog
index adeadf40..92285b8e 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,1269 +1,1273 @@
-swh-deposit (0.14.2-1~swh1~bpo10+1) buster-swh; urgency=medium
+swh-deposit (0.14.3-1~swh1) unstable-swh; urgency=medium
- * Rebuild for buster-swh
+ * New upstream release 0.14.3 - (tagged by Valentin Lorentz
+ <vlorentz@softwareheritage.org> on 2021-08-12 12:20:51 +0200)
+ * Upstream changes: - v0.14.3 - * Fix TypeError when accessing
+ the API root - * deposit_update_status: Add support for updating
+ status_detail
- -- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 22 Jun 2021 11:10:21 +0000
+ -- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 12 Aug 2021 10:26:17 +0000
swh-deposit (0.14.2-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.14.2 - (tagged by Antoine Lambert
<anlambert@softwareheritage.org> on 2021-06-22 13:03:32 +0200)
* Upstream changes: - version 0.14.2
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 22 Jun 2021 11:08:05 +0000
swh-deposit (0.14.1-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.14.1 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-05-26 09:26:57
+0200)
* Upstream changes: - v0.14.1 - Deal properly with header
response
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 26 May 2021 07:34:04 +0000
swh-deposit (0.14.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.14.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-05-25 15:42:12
+0200)
* Upstream changes: - v0.14.0 - server: Open a paginated list
user deposits endpoint - client: Open `swh deposit list` cli to
allow user to list their deposits - swh/deposit/client: Add more
coverage and types - tests: Rename test_deposit_list.py to
test_deposit_private_list.py
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 25 May 2021 13:54:40 +0000
swh-deposit (0.13.7-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.13.7 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-05-20 15:38:05
+0200)
* Upstream changes: - v0.13.7 - Fix crash on 'PUT Col-IRI'
- Stop setting MetadataAuthority.metadata and
MetadataFetcher.metadata - requirements-test: don't install
pytest-postgresql 3.0+
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 20 May 2021 13:44:13 +0000
swh-deposit (0.13.6-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.13.6 - (tagged by Antoine Lambert
<antoine.lambert@inria.fr> on 2021-04-29 14:23:04 +0200)
* Upstream changes: - version 0.13.6
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 29 Apr 2021 12:29:05 +0000
swh-deposit (0.13.5-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.13.5 - (tagged by Valentin Lorentz
<vlorentz@softwareheritage.org> on 2021-04-08 14:59:58 +0200)
* Upstream changes: - v0.13.5 - * Reorganize documentation
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 08 Apr 2021 13:11:11 +0000
swh-deposit (0.13.4-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.13.4 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-03-26 18:02:42
+0100)
* Upstream changes: - v0.13.4 - docs: Add a reference
authentication page to explain auth schemes - docs/sys-info:
Update deployment documentation - docs/sys-info: Update
information and rework sentence phrasing - docs: Unify READMEs
in the documentation and the source code - cli.client: Fix
sphinx indentation warning
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Fri, 26 Mar 2021 17:07:13 +0000
swh-deposit (0.13.3-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.13.3 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-03-24 16:49:18
+0100)
* Upstream changes: - v0.13.1 - deposit.auth: Namespace cache
keys per realm, client_id and user_id - deposit.auth: Logs
warning during cache token retrieval failure
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 24 Mar 2021 15:54:29 +0000
swh-deposit (0.13.2-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.13.2 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-03-23 17:52:22
+0100)
* Upstream changes: - v0.13.2 - deposit.auth: Fix
authentication failure corner case
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 23 Mar 2021 16:57:49 +0000
swh-deposit (0.13.1-1~swh2) unstable-swh; urgency=medium
* Fix runtime dependency
-- Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Tue, 23 Mar 2021 14:18:01 +0100
swh-deposit (0.13.1-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.13.1 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-03-23 11:34:02
+0100)
* Upstream changes: - v0.13.1 - deposit.auth: Adjust
authentication error message - deposit.cli: Fix service document
error when failing to retrieve it - deposit.cli: Fix cli parsing
issue when xml error is returned by server
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 23 Mar 2021 10:40:03 +0000
swh-deposit (0.13.0-1~swh2) unstable-swh; urgency=medium
* Fix build time dependency release
-- Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Mon, 22 Mar 2021 12:40:40 +0100
swh-deposit (0.13.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.13.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-03-22 12:03:21
+0100)
* Upstream changes: - v0.13.0 - Allow to configure
authentication mechanism per config file - Delegate
authentication to keycloak
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 22 Mar 2021 11:13:37 +0000
swh-deposit (0.12.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.12.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-03-16 09:28:08
+0100)
* Upstream changes: - v0.12.0 - Add deposit info to objects
added to swh-storage from metadata-only deposits - tests:
Simplify discovery_date comparison. - Check a SWHID exists in
the archive before accepting a metadata-only deposit
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 16 Mar 2021 08:32:08 +0000
swh-deposit (0.11.1-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.11.1 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-03-10 12:53:10
+0100)
* Upstream changes: - v0.11.1 - tests: Start testing migration
scripts
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 10 Mar 2021 11:56:53 +0000
swh-deposit (0.11.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.11.0 - (tagged by Valentin Lorentz
<vlorentz@softwareheritage.org> on 2021-03-02 12:33:11 +0100)
* Upstream changes: - v0.11.0 - * Use CoreSWHID/QualifiedSWHID
instead of the deprecated SWHID class.
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 02 Mar 2021 11:38:13 +0000
swh-deposit (0.10.2-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.10.2 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-02-26 12:08:20
+0100)
* Upstream changes: - v0.10.2 - deposit.urls: Retro-
compatibility fix about import and type conflict
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Fri, 26 Feb 2021 11:14:00 +0000
swh-deposit (0.10.1-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.10.1 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-02-25 16:51:28
+0100)
* Upstream changes: - v0.10.1 - tests: Tentatively try to fix
the debian build
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 25 Feb 2021 15:56:31 +0000
swh-deposit (0.10.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.10.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-02-25 15:52:55
+0100)
* Upstream changes: - v0.10.0 - Aggregate deposit archives
into a temporary tarball instead of a zip - Fix swh.deposit.urls
typing - deposit.cli: Warn users when missing origin tags are
detected - Stop recommending the Slug header as the alternative
to <external_identifier>. - deposit.client.cli: Expose --create-
origin flag and deprecate --slug - test: Fix failing test
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 25 Feb 2021 14:58:13 +0000
swh-deposit (0.9.2-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.9.2 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-01-14 12:42:12
+0100)
* Upstream changes: - v0.9.2 - Make <swh:add_to_origin> set
Deposit.origin_url + disable aggressive deprecation warning -
test_collection_reuse_slug: Assert Deposit.origin_url is set. -
cli: Drop dead code - user-manual: Add deposit metadata update
scenario - docs: Explicit the new deposit creation
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 14 Jan 2021 11:46:11 +0000
swh-deposit (0.9.1-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.9.1 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-01-06 09:34:41
+0100)
* Upstream changes: - v0.9.1 - client: Fix url to update
metadata on deposit with status 'done' - doc: Add cli section
- homepage: Fix broken link
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 06 Jan 2021 08:39:41 +0000
swh-deposit (0.9.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.9.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2021-01-04 17:31:18
+0100)
* Upstream changes: - v0.9.0 - Handle <swh:create_origin> /
<swh:add_to_origin> in multipart uploads - Split
<swh:create_origin> / <swh:add_to_origin> handling to its own
function - Catch invalid dates before marking a deposit as
verified - tests: Reorganize and refactor boilerplate -
docs: Rephrase introduction of the metadata-only deposit
documentation - docs: Document metadata updates
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 05 Jan 2021 08:06:42 +0000
swh-deposit (0.8.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.8.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-12-18 10:20:12
+0100)
* Upstream changes: - v0.8.0 - Allow metadata only deposit
(server, client) - Documentation updated accordingly
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Fri, 18 Dec 2020 09:30:06 +0000
swh-deposit (0.7.3-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.7.3 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-12-15 15:09:39
+0100)
* Upstream changes: - v0.7.3 - server: Fix metadata-only
deposit which are currently rejected - Allow metadata-only
deposit client side - Trap and report exceptions in a unified
way within the cli - Move parse_swh_reference to
swh.deposit.utils namespace
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 15 Dec 2020 14:13:15 +0000
swh-deposit (0.7.2-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.7.2 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-12-10 19:09:54
+0100)
* Upstream changes: - v0.7.2 - swh/deposit/migrations/0021:
Fix migration script
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 10 Dec 2020 18:13:50 +0000
swh-deposit (0.7.1-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.7.1 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-12-10 13:55:22
+0100)
* Upstream changes: - v0.7.1 - Use string equality instead of
substring search to check for mandatory fields. - Accept
<codemeta:name> and <codemeta:author> as alternatives to
<atom:name>/<atom:title> and <atom:author>.
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 10 Dec 2020 13:01:47 +0000
swh-deposit (0.7.0-1~swh2) unstable-swh; urgency=medium
* Bump new dependency
-- Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Tue, 08 Dec 2020 05:12:30 +0000
swh-deposit (0.7.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.7.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-12-08 13:25:57
+0100)
* Upstream changes: - v0.7.0 - docs: Add a complete deposit
protocol reference - Implement tag <swh:add_to_origin> to
replace the Slug header for parent - relationships -
Implement tag <swh:create_origin> to replace the Slug header -
Return the origin url on 'GET State-IRI' - Make the Slug header
optional - docs: Improve documentations and drop non implemented
specifications - Remove the <client> tag from the protocol. -
Remove the <external_identifier> tag from the protocol. - Move
SWH-specific tags to the
https://www.softwareheritage.org/schema/2018/deposit namespace -
logging: Log error messages when that occurs - typing: Improve
deposit types - Refactor exception handling - Split SE-IRI
and Edit-IRI. - remove assumption that Edit-IRI and SE-IRI are
the same from test - Rename files and classes in
swh/deposit/api/deposit_* to be consistent with SWORD terminology.
- swh.xsd: Use the
https://www.softwareheritage.org/schema/2018/deposit namespace -
Fix SWORD XMLNS (http://purl.org/net/sword/ ->
http://purl.org/net/sword/terms/) - Refactoring to clarify
source - tests: Refactoring
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 08 Dec 2020 12:35:45 +0000
swh-deposit (0.6.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.6.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-11-18 15:58:34
+0100)
* Upstream changes: - v0.6.0 - Adapt existing POST to a
collection to allow metadata-only deposit -
deposit.settings.production: Remove the logging django configuration
- Customize the user-agent header in deposit client classes -
Simplify `swh deposit upload` cli options - doc: improve the
user manual documentation - doc: rename Getting Started as User
Manual and update the content - doc: add an introduction
paragraph in blueprint.rst - doc: improve the spec-loading doc
- doc: improve the doc of API endpoints - doc: rename
docs/specs/specs.rst as docs/specs/index.rst - doc: Add a
description of the deposit in the docs' index page - doc: spec-
meta-deposit: Update metadata-only deposit samples and requirements
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 18 Nov 2020 15:01:09 +0000
swh-deposit (0.5.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.5.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-10-19 09:53:36
+0200)
* Upstream changes: - v0.5.0 - tests: Migrate configuration to
the latest schema change - spec-technical: Add 'partial' self-
loop + annotate transitions in the status diagram - spec-
technical: Rewrite state diagram using Plantuml.
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 19 Oct 2020 07:58:12 +0000
swh-deposit (0.4.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.4.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-10-16 08:57:58
+0200)
* Upstream changes: - v0.4.0 - cli.client: Adapt metadata
generation so server side checks pass - Update blueprints with
correct status names and definitions - Rewrite blueprint
flowcharts using Plantuml instead of .png files. -
deposit.cli.admin: Add coverage on `swh deposit reschedule` cli
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Fri, 16 Oct 2020 07:02:23 +0000
swh-deposit (0.3.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.3.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-10-13 15:12:50
+0200)
* Upstream changes: - v0.3.0 - deposit_client: Allow deposit
metadata update on completed deposit - deposit.client: Improve
cli error messages and add missing coverage - cli.client: Add
coverage - cli.client: Add types - cli.client: Add types and
refactor tests - conftest: Declare swh.core pytest_plugin -
deposit: Reuse config.load_from_envvar for configuration loading
- test_deposit_content: Add missing coverage
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 13 Oct 2020 13:16:30 +0000
swh-deposit (0.2.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.2.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-10-01 15:25:42
+0200)
* Upstream changes: - v0.2.0 - Allow deposit metadata update
on deposit already completed - Transit raw metadata to the
loader to unify with metadata update scenario - deposit*: Rename
internally swh_id references to swhid - deposit.parsers: Process
namespace when using xmltodict.parse - tests: Add missing update
scenarios - tests: Explicit the bad request scenario error
messages - tests: Ensure all empty body test cases are covered
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 01 Oct 2020 13:29:29 +0000
swh-deposit (0.1.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.1.0 - (tagged by David Douard
<david.douard@sdfa3.org> on 2020-09-25 11:49:24 +0200)
* Upstream changes: - v0.1.0
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Fri, 25 Sep 2020 09:53:06 +0000
swh-deposit (0.0.90-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.90 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-06-01 12:25:53
+0200)
* Upstream changes: - v0.0.90 - swh.deposit.models: Upload
deposit archives to dedicated folder
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 01 Jun 2020 10:31:57 +0000
swh-deposit (0.0.89-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.89 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-20 17:45:23
+0200)
* Upstream changes: - v0.0.89 - private/deposit_list: Allow
exclusion patterns from listing
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 20 May 2020 15:49:10 +0000
swh-deposit (0.0.88-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.88 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-20 11:21:31
+0200)
* Upstream changes: - v0.0.88 - Drop swh_anchor_id* columns
from Deposit model
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 20 May 2020 09:28:14 +0000
swh-deposit (0.0.87-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.87 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-15 12:48:31
+0200)
* Upstream changes: - v0.0.87 - Migrate deposit SWHIDs (data)
to the new specification - Update deposit swhid to respect the
latest specification update - Fix typos
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Fri, 15 May 2020 10:52:33 +0000
swh-deposit (0.0.86-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.86 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-11 18:04:03
+0200)
* Upstream changes: - v0.0.86 - origin/master Make deposit
client deal properly with maintenance issues - maintenance
exception: Make the exception appear as raw content
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 11 May 2020 16:07:28 +0000
swh-deposit (0.0.85-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.85 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-11 15:29:11
+0200)
* Upstream changes: - v0.0.85 - origin/master Align 503
exceptions output format with existing errors - Add deposit
exception handler to improve default error display - tox: Drop
django1 entry, we use django2 in production - settings.common:
Format
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 11 May 2020 13:40:23 +0000
swh-deposit (0.0.84-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.84 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-07 14:48:54
+0100)
* Upstream changes: - v0.0.84 - test: Add checks scenario test
cases - test_task: Mark some test to be ignored during debian
build
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 07 May 2020 13:52:51 +0000
swh-deposit (0.0.83-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.83 - (tagged by Valentin Lorentz
<vlorentz@softwareheritage.org> on 2020-05-07 11:52:28 +0200)
* Upstream changes: - v0.0.83 - * Pass collection + id to the
checker instead of an URL.
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 07 May 2020 09:56:40 +0000
swh-deposit (0.0.82-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.82 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2020-04-23 16:40:34
+0200)
* Upstream changes: - v0.0.82 - deposit_read: Simplify api to
return only relevant deposit information - setup: Update the
minimum required runtime python3 version - spec: reference SWHID
using explicit anchors - Update type annotations and signatures
to match djangorestframework-stubs - pytest.ini: Avoid loading
flask plugin to prevent fixture name clash - Add a
pyproject.toml file to target py37 for black - Enable black -
tests: Adapt init_sentry api change call with environment parameter
- docs: Fix sphinx warnings - tests/gunicorn_config: Fix tests
after recent changes in swh-core
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 23 Apr 2020 14:45:49 +0000
swh-deposit (0.0.81-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.81 - (tagged by David Douard
<david.douard@sdfa3.org> on 2020-01-13 11:50:07 +0100)
* Upstream changes: - v0.0.81
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 13 Jan 2020 11:07:54 +0000
swh-deposit (0.0.80-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.80 - (tagged by David Douard
<david.douard@sdfa3.org> on 2020-01-09 15:54:57 +0100)
* Upstream changes: - v0.0.80
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 09 Jan 2020 15:00:32 +0000
swh-deposit (0.0.79-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.79 - (tagged by Valentin Lorentz
<vlorentz@softwareheritage.org> on 2019-12-19 17:37:41 +0100)
* Upstream changes: - v0.0.79 - * requirements: Pin mypy and
django-stubs version - * homepage: Improve sentence phrasing
- * deposit.api: Add a basic api page to avoid broken link - *
removed dead and deprecated code - * Add sentry integration.
- * Fix log level + status code of the client CLI in case of error.
- * Improve validation of --author and --name. - * Update
documentation of --author to use names instead of emails.
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 19 Dec 2019 16:42:28 +0000
swh-deposit (0.0.78-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.78 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-11-25 18:28:13
+0100)
* Upstream changes: - v0.0.78 - deposit.signal: Simplify
configuration entry
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 25 Nov 2019 17:34:33 +0000
swh-deposit (0.0.77-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.77 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-11-25 14:48:59
+0100)
* Upstream changes: - v0.0.77 - deposit.signals: Send
versioned scheduler tasks - deposit.signals: Scheduler load-
deposit task with new endpoints - mypy: Fix missing import
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 25 Nov 2019 14:02:38 +0000
swh-deposit (0.0.76-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.76 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-11-25 12:26:07
+0100)
* Upstream changes: - v0.0.76 - Start adding mypy annotation
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 25 Nov 2019 11:29:48 +0000
swh-deposit (0.0.75-1~swh2) unstable-swh; urgency=medium
* Add egg-info to pybuild.testfiles
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 20 Nov 2019 15:06:31 +0100
swh-deposit (0.0.75-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.75 - (tagged by Nicolas Dandrimont
<nicolas@dandrimont.eu> on 2019-10-30 16:50:25 +0100)
* Upstream changes: - Release swh.deposit v0.0.75 - Revert
changes to the task signature until the new loader is ready -
Migrate tests to pytest
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 30 Oct 2019 15:54:55 +0000
swh-deposit (0.0.74-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.74 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-10-09 11:30:00
+0200)
* Upstream changes: - v0.0.74 - deposit.signals: Scheduler
load-deposit task with new endpoints - deposit.private.api:
Expose new endpoints with no collection name - models: Migrate
model to enforce check on delete - setup: register the worker
task in the swh.workers entrypoint - tests: Explicit private
tests in their names - admin CLI: avoid redefining deposit name
in admin subcommand
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 09 Oct 2019 09:35:03 +0000
swh-deposit (0.0.73-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.73 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-09-05 10:15:29
+0200)
* Upstream changes: - v0.0.73 - loader: Add missing visit_type
attribute. - cli/client: Simplify url definition to use -
cli/admin: Add the default domain value to empty - deposit_data:
Remove no longer used DepositRequestType references - doc/sys-
info: Clarify commands - docs: add code of conduct document
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 05 Sep 2019 08:19:17 +0000
swh-deposit (0.0.72-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.72 - (tagged by Valentin Lorentz
<vlorentz@softwareheritage.org> on 2019-06-18 17:15:53 +0200)
* Upstream changes: - Remove argument origin_id from call to
Loader.send_origin_metadata. - It no longer needs that argument.
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 19 Jun 2019 14:26:44 +0000
swh-deposit (0.0.71-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.71 - (tagged by Antoine Lambert
<antoine.lambert@inria.fr> on 2019-05-23 11:02:05 +0200)
* Upstream changes: - version 0.0.71
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 23 May 2019 09:09:54 +0000
swh-deposit (0.0.70-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.70 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-05-10 12:04:33
+0200)
* Upstream changes: - v0.0.70 - Update documentations (getting-
started, metadata) - Improve cli client (expose new status
subcommand, clarify help messages) - Fixes some issues
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Fri, 10 May 2019 10:33:19 +0000
swh-deposit (0.0.69-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.69 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-05-09 10:25:43
+0200)
* Upstream changes: - v0.0.69 - cli.admin: Add an admin
subcommand 'deposit reschedule' - models: Keep scheduler task
ids reference on deposit model - cli: make the deposit cli
command a subcommand of the main 'swh' one - docs: update the
getting-started document to use 'swh deposit upload' command -
docs: Update the sys-info document to expose the `swh deposit admin
deposit - reschedule` subcommand
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 09 May 2019 08:35:53 +0000
swh-deposit (0.0.68-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.68 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-04-17 15:39:52
+0200)
* Upstream changes: - v0.0.68
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 17 Apr 2019 13:50:21 +0000
swh-deposit (0.0.67-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.67 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-02-20 17:39:59
+0100)
* Upstream changes: - v0.0.67 - settings.prod: Use
SWH_CONFIG_FILENAME to load & check swh config - requirements-
swh.txt: Update minimal version required for loader-tar - remove
debian/ tree from master branch
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 20 Feb 2019 16:45:22 +0000
swh-deposit (0.0.66-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.66 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-02-16 10:22:30
+0100)
* Upstream changes: - v0.0.66 - deposit.loader.checker: Fix
logger initialization
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Sat, 16 Feb 2019 09:27:54 +0000
swh-deposit (0.0.65-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.65 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-02-14 18:28:48
+0100)
* Upstream changes: - v0.0.65 - swh/manage: Fix flake8 warning
- Bump dependency on swh-scheduler 0.0.39 - Rewrite celery tasks
as a decorated function - loader.scheduler: Remove non
production code - deposit.loader.tasks: Add tests
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 14 Feb 2019 17:34:09 +0000
swh-deposit (0.0.63-1~swh2) unstable-swh; urgency=medium
* New upstream release, fixing dependencies
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 14 Feb 2019 18:16:04 +0100
swh-deposit (0.0.63-1~swh1) unstable-swh; urgency=medium
* v0.0.63
* deposit_list: Return status_detail as string message and not as
nested
* dict
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 17 Sep 2018 16:23:58 +0200
swh-deposit (0.0.62-1~swh1) unstable-swh; urgency=medium
* v0.0.62
* private.deposit_list: Make the endpoint private
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 12 Sep 2018 15:58:21 +0200
swh-deposit (0.0.61-1~swh1) unstable-swh; urgency=medium
* v0.0.61
* Add api endpoint to list deposits with pagination
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 12 Sep 2018 14:44:58 +0200
swh-deposit (0.0.60-1~swh1) unstable-swh; urgency=medium
* v0.0.60
* Fix production issue regarding deposit status endpoint.
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 04 Sep 2018 18:30:01 +0200
swh-deposit (0.0.59-1~swh1) unstable-swh; urgency=medium
* v0.0.59
* deposit.utils: Fix the potential metadata information loss
* docs: Add sparse/metadata deposit specs
* docs: Update documentation about persistent id with context
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 24 Jul 2018 14:15:55 +0200
swh-deposit (0.0.58-1~swh1) unstable-swh; urgency=medium
* v0.0.58
* d/*: Update to latest python3-swh.model dependency version
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 23 Jul 2018 14:31:46 +0200
swh-deposit (0.0.57-1~swh1) unstable-swh; urgency=medium
* v0.0.57
* swh.deposit.client: Simplify client parsing
* api/deposit_status: Make swh-id be a directory id derivative
* swh.deposit.models: Keep deposit request's raw metadata
* bin: Migrate internal script to use the deposit client
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 23 Jul 2018 13:43:23 +0200
swh-deposit (0.0.56-1~swh1) unstable-swh; urgency=medium
* v0.0.56
* docs: Update deposit with status rejected documentation
* deposit_status: Update the deposit status endpoint with details for
* rejected deposit
* deposit_check: Reject invalid deposit (associated archive containing
* only single archive)
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 17 Jul 2018 13:27:35 +0200
swh-deposit (0.0.55-1~swh1) unstable-swh; urgency=medium
* v0.0.55
* deposit_status: Open detailed status when a deposit fails the checks
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 11 Jul 2018 12:06:10 +0200
swh-deposit (0.0.54-1~swh1) unstable-swh; urgency=medium
* v0.0.54
* deposit_check: Improve details in failing checks
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 10 Jul 2018 10:23:35 +0200
swh-deposit (0.0.53-1~swh1) unstable-swh; urgency=medium
* v0.0.53
* swh.deposit.parsers: Fix xml parsing to not lose duplicated entries
* swh.deposit.tests: Make sure failing tests are complete
* deposit_update: Fix check error during update with wrong mimetype
* deposit_read: Persistent identifier representation has changed
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 06 Jul 2018 16:26:08 +0200
swh-deposit (0.0.52-1~swh1) unstable-swh; urgency=medium
* v0.0.52
* Make the deposit's scheduler configuration adjustable
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 03 May 2018 15:16:30 +0200
swh-deposit (0.0.51-1~swh1) unstable-swh; urgency=medium
* v0.0.51
* Improve origin_visit initialization step
* Properly sandbox the prepare statement so that if it breaks, we can
* update appropriately the visit with the correct status
* Let the visit date be set in lower layer
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 07 Mar 2018 11:07:11 +0100
swh-deposit (0.0.50-1~swh1) unstable-swh; urgency=medium
* v0.0.50
* Bump requirements up for new swh.loader.tar
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 12 Feb 2018 11:21:03 +0100
swh-deposit (0.0.49-1~swh1) unstable-swh; urgency=medium
* Release swh.deposit v0.0.49
* Use snapshots instead of occurrences
-- Nicolas Dandrimont <nicolas@dandrimont.eu> Tue, 06 Feb 2018 14:48:53 +0100
swh-deposit (0.0.48-1~swh1) unstable-swh; urgency=medium
* v0.0.48
* swh.deposit.api.private: Fix revision message missing client name
* docs: simplify endpoints and delete all sword text
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 02 Feb 2018 08:37:39 +0100
swh-deposit (0.0.47-1~swh1) unstable-swh; urgency=medium
* v0.0.47
* swh.loader: Be consistent in returning loader result in task
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 30 Jan 2018 19:03:49 +0100
swh-deposit (0.0.46-1~swh1) unstable-swh; urgency=medium
* v0.0.46
* swh.deposit.client: Explicit private api client
* swh.deposit.client.cli: Fix flag compatibility issue
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 30 Jan 2018 12:19:09 +0100
swh-deposit (0.0.45-1~swh1) unstable-swh; urgency=medium
* v0.0.45
* Simplify collection name retrieval
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 29 Jan 2018 18:09:29 +0100
swh-deposit (0.0.44-1~swh1) unstable-swh; urgency=medium
* v0.0.44
* docs: Update getting-started documentations
* swh.deposit.api: Add swh_id key to None by default in status
endpoint
* swh.deposit.api: Fix tar archive permission in update endpoints
* swh.deposit.api.private: Fix pep8 violation about catch Exception
* swh.deposit.api: Do not hardcode the server uri in service document
* endpoint
* swh.deposit.client: Add a deposit client
* d/control: Create python3-swh.deposit.client package
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 29 Jan 2018 17:41:12 +0100
swh-deposit (0.0.43-1~swh1) unstable-swh; urgency=medium
* v0.0.43
* swh.deposit.api: Deposit returns persistent identifiers
* swh.deposit.api: Rename deposit statuses
* swh.deposit: Support standard tarball formats (.tar.*)
* docs: Update documentation about new archive format support
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 17 Jan 2018 12:02:20 +0100
swh-deposit (0.0.42-1~swh1) unstable-swh; urgency=medium
* v0.0.42
* Fix cosmetic issues in deposit.s.o
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 10 Jan 2018 16:25:48 +0100
swh-deposit (0.0.41-1~swh1) unstable-swh; urgency=medium
* v0.0.41
* swh.deposit.checks: Add url validation
* swh.deposit: Add splash screen to homepage
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 10 Jan 2018 12:12:43 +0100
swh-deposit (0.0.40-1~swh1) unstable-swh; urgency=medium
* v0.0.40
* Fix corner case on deposit checks and clarify intents on checks
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 08 Jan 2018 18:21:38 +0100
swh-deposit (0.0.39-1~swh1) unstable-swh; urgency=medium
* v0.0.39
* Amend detail status message
* docs: Fix url
* Fix check of deposit without content scenario
* Refactor metadata check with pythonic function
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 08 Jan 2018 12:43:15 +0100
swh-deposit (0.0.38-1~swh1) unstable-swh; urgency=medium
* v0.0.38
* Provide existing swh id in the status api when it exists
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 08 Dec 2017 09:41:21 +0100
swh-deposit (0.0.37-1~swh1) unstable-swh; urgency=medium
* v0.0.37
* Adapt to latest dependency on loader-core and storage
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 07 Dec 2017 15:30:32 +0100
swh-deposit (0.0.36-1~swh1) unstable-swh; urgency=medium
* v0.0.36
* swh.deposit.api.private: Deposit's author is swh
* d/control: Bump to latest dependencies version
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 06 Dec 2017 12:21:45 +0100
swh-deposit (0.0.35-1~swh1) unstable-swh; urgency=medium
* v0.0.35
* swh.deposit.loader: Fix intermediary status
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 05 Dec 2017 19:23:40 +0100
swh-deposit (0.0.34-1~swh1) unstable-swh; urgency=medium
* v0.0.34
* d/control: Bump to latest version
* Fix client config typo for local url
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 05 Dec 2017 15:43:59 +0100
swh-deposit (0.0.33-1~swh1) unstable-swh; urgency=medium
* v0.0.33
* Bump to latest swh.loader.tar
* dev swh.deposit.api: Add parent deposit to deposit at creation time
* swh.deposit.service: Clean dead code
* bin/Makefile: Add multipart deposit sample script
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 04 Dec 2017 18:59:50 +0100
swh-deposit (0.0.32-1~swh1) unstable-swh; urgency=medium
* v0.0.32
* Migrate swh.deposit.injection module to swh.deposit.loader
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 30 Nov 2017 16:27:19 +0100
swh-deposit (0.0.31-1~swh1) unstable-swh; urgency=medium
* v0.0.31
* swh.deposit.api: Separate public/private api
* swh.deposit.api.common: Fix authentication issue in production
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 30 Nov 2017 13:18:15 +0100
swh-deposit (0.0.30-1~swh1) unstable-swh; urgency=medium
* v0.0.30
* swh.deposit.signals: Fix wrong task scheduling argument
* swh.deposit.migration: Remove default value
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 29 Nov 2017 18:34:29 +0100
swh-deposit (0.0.29-1~swh1) unstable-swh; urgency=medium
* v0.0.29
* Fix inconsistent term in code
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 29 Nov 2017 15:22:36 +0100
swh-deposit (0.0.28-1~swh1) unstable-swh; urgency=medium
* v0.0.28
* swh.deposit: Use revision_id as swh_id
* swh.deposit: Untangle checks from the current client/server requests
flow
* swh.deposit.injection: Trigger scheduling of checks on deposit
* swh.deposit.injection: Trigger scheduling of loading on deposit
* swh.deposit.injection: Add origin_metadata during loading
* d/control: Bump to latest swh layers
* swh.deposit.tests: Add and refactor tests
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 29 Nov 2017 15:03:56 +0100
swh-deposit (0.0.27-1~swh1) unstable-swh; urgency=medium
* v0.0.27
* swh.deposit.parsers: Fix edge case about decimal serialization
* swh.deposit.injection.loader: Add test on loading a deposit
* swh.loader.test: Fix path initialization
* swh.deposit.injection.scheduler: Adapt default task
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 17 Nov 2017 10:54:08 +0100
swh-deposit (0.0.26-1~swh1) unstable-swh; urgency=medium
* v0.0.26
* swh.deposit: Be consistent in the deposit_status key returned
* docs: Move actual docs inside the docs/ folder
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 25 Oct 2017 17:37:02 +0200
swh-deposit (0.0.25-1~swh1) unstable-swh; urgency=medium
* v0.0.25
* swh.deposit.production: Add support for proxy headers in django
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 24 Oct 2017 14:08:42 +0200
swh-deposit (0.0.24-1~swh1) unstable-swh; urgency=medium
* v0.0.24
* swh.deposit.api: Fix 500 error when browsing api through browser
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 23 Oct 2017 15:37:11 +0200
swh-deposit (0.0.23-1~swh1) unstable-swh; urgency=medium
* v0.0.23
* swh.deposit.api: Deal with new 'rejected' status on deposit
* docs: Update documentation to latest development
* swh.deposit.api: Update docstrings properly
* swh.deposit.api: Add post check validation on deposit
* swh.deposit.tests: Add edge case scenario tests (upload size limit,
* deposit read archives, etc...)
* swh.deposit.api: Fix mismatch hash check message
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 23 Oct 2017 11:31:39 +0200
swh-deposit (0.0.22-1~swh1) unstable-swh; urgency=medium
* v0.0.22
* swh.deposit.api: Return fqdn urls
* swh.deposit.api: Use variable to define the pivot status 'ready'
* swh.deposit.api: Add state iri in the deposit receipt
* swh.deposit.api: Add deposit's status in the deposit receipt
* swh.deposit.api: Explicit no support for Metadata-Relevant header
* swh.deposit.tests: Fix potential listing error
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 19 Oct 2017 15:38:01 +0200
swh-deposit (0.0.21-1~swh1) unstable-swh; urgency=medium
* v0.0.21
* swh.deposit.api: Simplify clean up temporary directory routine for
* deposit read archive api
* swh.deposit.tests: Add missing test cases around deposit read api
* README-injection: Improve sentence phrasing
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 18 Oct 2017 11:36:40 +0200
swh-deposit (0.0.20-1~swh1) unstable-swh; urgency=medium
* v0.0.20
* swh.deposit.scheduler: Move scheduling part to swh.deposit.injection
* swh.deposit.injection: Separation of concern between reading/loading
* swh.deposit.api: Remove dead code
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 16 Oct 2017 18:23:19 +0200
swh-deposit (0.0.19-1~swh1) unstable-swh; urgency=medium
* v0.0.19
* Define summary message for method not allowed endpoints
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 16 Oct 2017 12:46:11 +0200
swh-deposit (0.0.18-1~swh1) unstable-swh; urgency=medium
* v0.0.18
* swh.deposit.api.deposit: make slug header mandatory
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Sat, 14 Oct 2017 10:45:08 +0200
swh-deposit (0.0.17-1~swh1) unstable-swh; urgency=medium
* v0.0.17
* Fix missing files for packaging
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 13 Oct 2017 16:37:27 +0200
swh-deposit (0.0.16-1~swh1) unstable-swh; urgency=medium
* v0.0.16
* packaging: Split python3-swh.deposit / python3-swh.deposit.injection
* swh.deposit.injection: Add deposit archive ingestion task
* packaging: Cleanup
* swh.deposit.auth: Cleanup authentication to use directly drf's
* swh.deposit.api: Split between private and public api
* swh.deposit.api: Add private api to update deposit's status
* swh.deposit.scheduler.cli: Add one-shot task scheduling machinery
* swh.deposit.tests: use the collection name when creating uri
* swh.deposit.admin: Clean up unused code
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 13 Oct 2017 15:41:47 +0200
swh-deposit (0.0.15-1~swh1) unstable-swh; urgency=medium
* v0.0.15
* swh.deposit.api: Add service to clean up temporary archives
* swh.deposit.api: Add private api to read a deposit's raw content
* swh.deposit.api: Update docstring
* swh.deposit.api: Switch from objstorage layer to django's
* docs: Fix typo in private yaml sample
* README-dev: Update documentation about bootstraping the dev env
* swh.deposit.tests: Check existence before directory cleanup
* Remove reference to noop and verbose since no longer in spec 2.0
* bin: Reference sample executables to exercise local run
* swh.deposit.scheduler.cli: Add a scheduling implementation on
deposit
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 09 Oct 2017 10:23:48 +0200
swh-deposit (0.0.14-1~swh1) unstable-swh; urgency=medium
* v0.0.14
* swh.deposit.tests: Add missing test cases scenario about updates
* docs: Improve and make the documentation browsable through browser
* docs: Add README-sys, README-getting-started, README-injection
* swh.deposit.create_user: Fix collection setup
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 28 Sep 2017 16:56:54 +0200
swh-deposit (0.0.13-1~swh1) unstable-swh; urgency=medium
* v0.0.13
* swh.deposit.api: Restrict access to one's own collection
* swh.deposit.api: Unify checks on all endpoints
* swh.deposit: Separate the collection from the client notion
* swh.deposit.api: Add delete deposit endpoint
* swh.deposit.api: Add delete content (archives) from deposit
* swh.deposit.api: Empty post on EDIT-IRI can finalize a deposit
* swh.deposit.model: Relax unicity constraint on external id
* swh.deposit.api: PUT does not permit to have the deposit_id None
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 25 Sep 2017 18:02:46 +0200
swh-deposit (0.0.12-1~swh1) unstable-swh; urgency=medium
* v0.0.12
* swh.deposit.api: Separate the replace metadata from the replace
* archive routine
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 22 Sep 2017 18:59:48 +0200
swh-deposit (0.0.11-1~swh1) unstable-swh; urgency=medium
* v0.0.11
* swh.deposit.static: Add static folder
* swh.deposit.api: Accept modifications to deposit only in partial
status
* swh.deposit.api: Update/Add new deposit metadata/archive
* swh.deposit.api.tests: Tests new use cases
* README: Update and simplify documentation
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 22 Sep 2017 16:24:25 +0200
swh-deposit (0.0.10-1~swh1) unstable-swh; urgency=medium
* v0.0.10
* swh.deposit.config: Centralize default config in .config module
* swh.deposit.api: Split api module definition
* swh.deposit.auth: Improve white-listing mechanism
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 21 Sep 2017 10:41:04 +0200
swh-deposit (0.0.9-1~swh1) unstable-swh; urgency=medium
* v0.0.9
* swh.deposit.api: white list / from authentication
* swh.deposit.api: Update state iri endpoint
* swh.deposit.api: Update new IRI endpoints to deal with update
* swh.deposit.api.deposit: Clean up dead code
* README: Update specification on IRIs
* swh.deposit.urls: Fix endpoints to finish with trailing /
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 20 Sep 2017 18:12:33 +0200
swh-deposit (0.0.8-1~swh1) unstable-swh; urgency=medium
* v0.0.8
* swh.deposit.settings: Split logging configuration per platform
* swh.deposit.settings.common: Prefer configuration over code
* swh.deposit.api: Enforce basic authentication
* swh.deposit: Clarify SWHDefaultConfig class's intent
* clean up: Removing user test api endpoints
* doc: Update docstrings
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 19 Sep 2017 14:23:05 +0200
swh-deposit (0.0.7-1~swh1) unstable-swh; urgency=medium
* v0.0.7
* Add swh.deposit.create_user routine to setup user information
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 19 Sep 2017 10:44:59 +0200
swh-deposit (0.0.6-1~swh1) unstable-swh; urgency=medium
* v0.0.6
* Make the packages include all that's needed (templates, fixtures)
* Fix typo in production settings file
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 18 Sep 2017 16:17:53 +0200
swh-deposit (0.0.5-1~swh1) unstable-swh; urgency=medium
* v0.0.5
* Package and add missing python3-djangorestframework-xml
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 18 Sep 2017 15:10:01 +0200
swh-deposit (0.0.4-1~swh1) unstable-swh; urgency=medium
* v0.0.4
* README: Update spec documentation according to latest development
* README-dev: Initiate a development readme to explicit the local
* dev/production mode
* swh.deposit.settings: Split profile configuration per deployment
* platform (dev, production)
* swh.deposit.views:
* Adapt returned errors to be sword compliant
* Change starting api route endpoints
* Improve deposit request headers checks
* swh.deposit.tests:
* Inhibit side-effects in tests (objstorage, configuration loading,
etc...)
* Add authentication in tests
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Sat, 16 Sep 2017 15:38:44 +0200
swh-deposit (0.0.3-1~swh1) unstable-swh; urgency=medium
* v0.0.3
* Migrate to django framework
* Deployment tryouts
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 01 Aug 2017 12:36:22 +0200
swh-deposit (0.0.2-1~swh1) unstable-swh; urgency=medium
* v0.0.2
* Fix db connection initialization
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 14 Jun 2017 13:49:31 +0200
swh-deposit (0.0.1-1~swh1) unstable-swh; urgency=medium
* Initial release
* v0.0.1
* Add basic server implementation for deployment testing
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 14 Jun 2017 12:48:37 +0200
diff --git a/swh.deposit.egg-info/PKG-INFO b/swh.deposit.egg-info/PKG-INFO
index 12406ca9..7c7e304d 100644
--- a/swh.deposit.egg-info/PKG-INFO
+++ b/swh.deposit.egg-info/PKG-INFO
@@ -1,98 +1,98 @@
Metadata-Version: 2.1
Name: swh.deposit
-Version: 0.14.2
+Version: 0.14.3
Summary: Software Heritage Deposit Server
Home-page: https://forge.softwareheritage.org/source/swh-deposit/
Author: Software Heritage developers
Author-email: swh-devel@inria.fr
License: UNKNOWN
Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest
Project-URL: Funding, https://www.softwareheritage.org/donate
Project-URL: Source, https://forge.softwareheritage.org/source/swh-deposit
Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-deposit/
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 5 - Production/Stable
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Provides-Extra: testing
Provides-Extra: server
License-File: LICENSE
License-File: AUTHORS
Software Heritage - Deposit
===========================
Simple Web-Service Offering Repository Deposit (S.W.O.R.D) is an interoperability
standard for digital file deposit.
This repository is both the `SWORD v2`_ Server and a deposit command-line client
implementations.
This implementation allows interaction between a client (a repository) and a server (SWH
repository) to deposit software source code archives and associated metadata.
Description
-----------
Most of the software source code artifacts present in the SWH Archive are gathered by
the mean of :term:`loader <loader>` workers run by the SWH project from sourve code
origins identified by :term:`lister <lister>` workers. This is a pull mechanism: it's
the responsibility of the SWH project to gather and collect source code artifacts that
way.
Alternatively, SWH allows its partners to push source code artifacts and metadata
directly into the Archive with a push-based mechanism. By using this possibility
different actors, holding software artifacts or metadata, can preserve their assets
without having to pass through an intermediate collaborative development platform, which
is already harvested by SWH (e.g GitHub, Gitlab, etc.).
This mechanism is the `deposit`.
The main idea is the deposit is an authenticated access to an API allowing the user to
provide source code artifacts -- with metadata -- to be ingested in the SWH Archive. The
result of that is a :ref:`SWHID <persistent-identifiers>` that can be used to uniquely
and persistently identify that very piece of source code.
This unique identifier can then be used to `reference the source code
<https://hal.archives-ouvertes.fr/hal-02446202>`_ (e.g. in a `scientific paper
<https://www.softwareheritage.org/2020/05/26/citing-software-with-style/>`_) and
retrieve it using the :ref:`vault <swh-vault>` feature of the SWH Archive platform.
The differences between a piece of code uploaded using the deposit rather than simply
asking SWH to archive a repository using the `save code now
<https://archive.softwareheritage.org/save/>`_ feature are:
- a deposited artifact is provided from one of the SWH partners which is regarded as a
trusted authority,
- a deposited artifact requires metadata properties describing the source code artifact,
- a deposited artifact has a codemeta_ metadata entry attached to it,
- a deposited artifact has the same visibility on the SWH Archive than a collected
repository,
- a deposited artifact can be searched with its provided url property on the SWH
Archive,
- the deposit API uses the `SWORD v2`_ API, thus requires some tooling to send deposits
to SWH. These tools are provided with this repository.
See the :ref:`deposit-user-manual` page for more details on how to use the deposit client
command line tools to push a deposit in the SWH Archive.
See the :ref:`deposit-api-specifications` reference pages of the SWORDv2 API implementation
in `swh.deposit` if you want to do upload deposits using HTTP requests.
Read the :ref:`deposit-metadata` chapter to get more details on what metadata
are supported when doing a deposit.
See :ref:`swh-deposit-dev-env` if you want to hack the code of the `swh.deposit` module.
See :ref:`swh-deposit-prod-env` if you want to deploy your own copy of the
`swh.deposit` stack.
.. _codemeta: https://codemeta.github.io/
.. _`SWORD v2`: http://swordapp.org/sword-v2/
diff --git a/swh/deposit/api/converters.py b/swh/deposit/api/converters.py
index 85943446..80b2012b 100644
--- a/swh/deposit/api/converters.py
+++ b/swh/deposit/api/converters.py
@@ -1,57 +1,65 @@
-# Copyright (C) 2017-2018 The Software Heritage developers
+# Copyright (C) 2017-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
def convert_status_detail(status_detail):
"""Given a status_detail dict, transforms it into a human readable string.
Dict has the following form (all first level keys are optional)::
{
'url': {
'summary': "summary-string",
'fields': [impacted-fields-list]
},
'metadata': [{
'summary': "summary-string",
'fields': [impacted-fields-list],
}],
'archive': [{
'summary': "summary-string",
'fields': [impacted-fields-list],
- }]
+ }],
+ 'loading': [
+ 'error 1',
+ 'error 2',
+ ],
}
Args:
status_detail (dict): The status detail dict with the syntax
mentioned
Returns:
the status detail as inlined string
"""
if not status_detail:
return None
def _str_fields(data):
fields = data.get("fields")
if not fields:
return ""
return " (%s)" % ", ".join(map(str, fields))
msg = []
for key in ["metadata", "archive"]:
_detail = status_detail.get(key)
if _detail:
for data in _detail:
msg.append("- %s%s\n" % (data["summary"], _str_fields(data)))
_detail = status_detail.get("url")
if _detail:
msg.append("- %s%s\n" % (_detail["summary"], _str_fields(_detail)))
+ _detail = status_detail.get("loading")
+ if _detail:
+ msg.extend(f"- {error}\n" for error in _detail)
+
if not msg:
return None
return "".join(msg)
diff --git a/swh/deposit/api/private/deposit_update_status.py b/swh/deposit/api/private/deposit_update_status.py
index cdec9773..df059261 100644
--- a/swh/deposit/api/private/deposit_update_status.py
+++ b/swh/deposit/api/private/deposit_update_status.py
@@ -1,115 +1,118 @@
# Copyright (C) 2017-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
from rest_framework.parsers import JSONParser
from swh.model.hashutil import hash_to_bytes
from swh.model.identifiers import CoreSWHID, ObjectType, QualifiedSWHID
from . import APIPrivateView
from ...errors import BAD_REQUEST, DepositError
from ...models import DEPOSIT_STATUS_DETAIL, DEPOSIT_STATUS_LOAD_SUCCESS, Deposit
from ..common import APIPut, ParsedRequestHeaders
MANDATORY_KEYS = ["origin_url", "revision_id", "directory_id", "snapshot_id"]
class APIUpdateStatus(APIPrivateView, APIPut):
"""Deposit request class to update the deposit's status.
HTTP verbs supported: PUT
"""
parser_classes = (JSONParser,)
def additional_checks(
self, request, headers: ParsedRequestHeaders, collection_name, deposit=None
):
"""Enrich existing checks to the default ones.
New checks:
- Ensure the status is provided
- Ensure it exists
- no missing information on load success update
"""
data = request.data
status = data.get("status")
if not status:
msg = "The status key is mandatory with possible values %s" % list(
DEPOSIT_STATUS_DETAIL.keys()
)
raise DepositError(BAD_REQUEST, msg)
if status not in DEPOSIT_STATUS_DETAIL:
msg = "Possible status in %s" % list(DEPOSIT_STATUS_DETAIL.keys())
raise DepositError(BAD_REQUEST, msg)
if status == DEPOSIT_STATUS_LOAD_SUCCESS:
missing_keys = []
for key in MANDATORY_KEYS:
value = data.get(key)
if value is None:
missing_keys.append(key)
if missing_keys:
msg = (
f"Updating deposit status to {status}"
f" requires information {','.join(missing_keys)}"
)
raise DepositError(BAD_REQUEST, msg)
return {}
def process_put(
self,
request,
headers: ParsedRequestHeaders,
collection_name: str,
deposit: Deposit,
) -> None:
"""Update the deposit with status and SWHIDs
Returns:
204 No content
400 Bad request if checks fail
"""
data = request.data
status = data["status"]
deposit.status = status
if status == DEPOSIT_STATUS_LOAD_SUCCESS:
origin_url = data["origin_url"]
directory_id = data["directory_id"]
revision_id = data["revision_id"]
dir_id = CoreSWHID(
object_type=ObjectType.DIRECTORY, object_id=hash_to_bytes(directory_id)
)
snp_id = CoreSWHID(
object_type=ObjectType.SNAPSHOT,
object_id=hash_to_bytes(data["snapshot_id"]),
)
rev_id = CoreSWHID(
object_type=ObjectType.REVISION, object_id=hash_to_bytes(revision_id)
)
deposit.swhid = str(dir_id)
# new id with contextual information
deposit.swhid_context = str(
QualifiedSWHID(
object_type=ObjectType.DIRECTORY,
object_id=hash_to_bytes(directory_id),
origin=origin_url,
visit=snp_id,
anchor=rev_id,
path="/",
)
)
else: # rejected
deposit.status = status
+ if "status_detail" in data:
+ deposit.status_detail = data["status_detail"]
+
deposit.save()
diff --git a/swh/deposit/client.py b/swh/deposit/client.py
index 50255f0f..41067f67 100644
--- a/swh/deposit/client.py
+++ b/swh/deposit/client.py
@@ -1,846 +1,849 @@
# Copyright (C) 2017-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
"""Module in charge of defining an swh-deposit client
"""
import hashlib
import logging
import os
from typing import Any, Dict, Optional, Tuple
from urllib.parse import urljoin
import warnings
import requests
from requests import Response
from requests.utils import parse_header_links
from swh.core.config import load_from_envvar
from swh.deposit import __version__ as swh_deposit_version
from swh.deposit.utils import parse_xml
logger = logging.getLogger(__name__)
def compute_unified_information(
collection: str,
in_progress: bool,
slug: str,
*,
filepath: Optional[str] = None,
swhid: Optional[str] = None,
**kwargs,
) -> Dict[str, Any]:
"""Given a filepath, compute necessary information on that file.
Args:
collection: Deposit collection
in_progress: do we finalize the deposit?
slug: external id to use
filepath: Path to the file to compute the necessary information out of
swhid: Deposit swhid if any
Returns:
dict with keys:
'slug': external id to use
'in_progress': do we finalize the deposit?
'content-type': content type associated
'md5sum': md5 sum
'filename': filename
'filepath': filepath
'swhid': deposit swhid
"""
result: Dict[str, Any] = {
"slug": slug,
"in_progress": in_progress,
"swhid": swhid,
}
content_type: Optional[str] = None
md5sum: Optional[str] = None
if filepath:
filename = os.path.basename(filepath)
md5sum = hashlib.md5(open(filepath, "rb").read()).hexdigest()
extension = filename.split(".")[-1]
if "zip" in extension:
content_type = "application/zip"
else:
content_type = "application/x-tar"
result.update(
{
"content-type": content_type,
"md5sum": md5sum,
"filename": filename,
"filepath": filepath,
}
)
return result
class MaintenanceError(ValueError):
"""Informational maintenance error exception
"""
pass
def handle_deprecated_config(config: Dict) -> Tuple[str, Optional[Tuple[str, str]]]:
warnings.warn(
'"config" argument is deprecated, please '
'use "url" and "auth" arguments instead; note that "auth" '
"expects now a couple (username, password) and not a dict.",
DeprecationWarning,
)
url: str = config["url"]
auth: Optional[Tuple[str, str]] = None
if config.get("auth"):
auth = (config["auth"]["username"], config["auth"]["password"])
return (url, auth)
class BaseApiDepositClient:
"""Deposit client base class
"""
def __init__(
self,
config: Optional[Dict] = None,
url: Optional[str] = None,
auth: Optional[Tuple[str, str]] = None,
):
if not url and not config:
config = load_from_envvar()
if config:
url, auth = handle_deprecated_config(config)
# needed to help mypy not be fooled by the Optional nature of url
assert url is not None
self.base_url = url.strip("/") + "/"
self.auth = auth
self.session = requests.Session()
if auth:
self.session.auth = auth
self.session.headers.update(
{"user-agent": f"swh-deposit/{swh_deposit_version}"}
)
def do(self, method, url, *args, **kwargs):
"""Internal method to deal with requests, possibly with basic http
authentication.
Args:
method (str): supported http methods as in self._methods' keys
Returns:
The request's execution
"""
full_url = urljoin(self.base_url, url.lstrip("/"))
return self.session.request(method, full_url, *args, **kwargs)
class PrivateApiDepositClient(BaseApiDepositClient):
"""Private API deposit client to:
- read a given deposit's archive(s)
- read a given deposit's metadata
- update a given deposit's status
"""
def archive_get(self, archive_update_url: str, archive: str) -> Optional[str]:
"""Retrieve the archive from the deposit to a local directory.
Args:
archive_update_url (str): The full deposit archive(s)'s raw content
to retrieve locally
archive (str): the local archive's path where to store
the raw content
Returns:
The archive path to the local archive to load.
Or None if any problem arose.
"""
response = self.do("get", archive_update_url, stream=True)
if response.ok:
with open(archive, "wb") as f:
for chunk in response.iter_content():
f.write(chunk)
return archive
msg = "Problem when retrieving deposit archive at %s" % (archive_update_url,)
logger.error(msg)
raise ValueError(msg)
def metadata_get(self, metadata_url):
"""Retrieve the metadata information on a given deposit.
Args:
metadata_url (str): The full deposit metadata url to retrieve
locally
Returns:
The dictionary of metadata for that deposit or None if any
problem arose.
"""
r = self.do("get", metadata_url)
if r.ok:
return r.json()
msg = "Problem when retrieving metadata at %s" % metadata_url
logger.error(msg)
raise ValueError(msg)
def status_update(
self,
update_status_url,
status,
+ status_detail=None,
revision_id=None,
directory_id=None,
origin_url=None,
):
"""Update the deposit's status.
Args:
update_status_url (str): the full deposit's archive
status (str): The status to update the deposit with
revision_id (str/None): the revision's identifier to update to
directory_id (str/None): the directory's identifier to update to
origin_url (str/None): deposit's associated origin url
"""
payload = {"status": status}
if revision_id:
payload["revision_id"] = revision_id
if directory_id:
payload["directory_id"] = directory_id
if origin_url:
payload["origin_url"] = origin_url
+ if status_detail:
+ payload["status_detail"] = status_detail
self.do("put", update_status_url, json=payload)
def check(self, check_url):
"""Check the deposit's associated data (metadata, archive(s))
Args:
check_url (str): the full deposit's check url
"""
r = self.do("get", check_url)
if r.ok:
data = r.json()
return data["status"]
msg = "Problem when checking deposit %s" % check_url
logger.error(msg)
raise ValueError(msg)
class BaseDepositClient(BaseApiDepositClient):
"""Base Deposit client to access the public api.
"""
def __init__(
self, config=None, url=None, auth=None, error_msg=None, empty_result={}
):
super().__init__(url=url, auth=auth, config=config)
self.error_msg = error_msg
self.empty_result = empty_result
def compute_url(self, *args, **kwargs):
"""Compute api url endpoint to query."""
raise NotImplementedError
def compute_method(self, *args, **kwargs):
"""Http method to use on the url"""
raise NotImplementedError
def parse_result_ok(
self, xml_content: str, headers: Optional[Dict] = None
) -> Dict[str, Any]:
"""Given an xml result from the api endpoint, parse it and returns a
dict.
"""
raise NotImplementedError
def compute_information(self, *args, **kwargs) -> Dict[str, Any]:
"""Compute some more information given the inputs (e.g http headers,
...)
"""
return {}
def parse_result_error(self, xml_content: str) -> Dict[str, Any]:
"""Given an error response in xml, parse it into a dict.
Returns:
dict with following keys:
'error': The error message
'detail': Some more detail about the error if any
"""
data = parse_xml(xml_content)
sword_error = data["sword:error"]
return {
"summary": sword_error["atom:summary"],
"detail": sword_error.get("detail", ""),
"sword:verboseDescription": sword_error.get("sword:verboseDescription", ""),
}
def do_execute(self, method: str, url: str, info: Dict, **kwargs) -> Response:
"""Execute the http query to url using method and info information.
By default, execute a simple query to url with the http method. Override this in
subclass to improve the default behavior if needed.
"""
return self.do(method, url, **kwargs)
def compute_params(self, **kwargs) -> Dict[str, Any]:
"""Determine the params out of the kwargs"""
return {}
def execute(self, *args, **kwargs) -> Dict[str, Any]:
"""Main endpoint to prepare and execute the http query to the api.
Raises:
MaintenanceError if some api maintenance is happening.
Returns:
Dict of computed api data
"""
url = self.compute_url(*args, **kwargs)
method = self.compute_method(*args, **kwargs)
info = self.compute_information(*args, **kwargs)
params = self.compute_params(**kwargs)
try:
response = self.do_execute(method, url, info, params=params)
except Exception as e:
msg = self.error_msg % (url, e)
result = self.empty_result
result.update(
{"error": msg,}
)
return result
else:
if response.ok:
if int(response.status_code) == 204: # 204 returns no body
return {"status": response.status_code}
else:
headers = dict(response.headers) if response.headers else None
return self.parse_result_ok(response.text, headers)
else:
error = self.parse_result_error(response.text)
empty = self.empty_result
error.update(empty)
if response.status_code == 503:
summary = error.get("summary")
detail = error.get("sword:verboseDescription")
# Maintenance error
if summary and detail:
raise MaintenanceError(f"{summary}: {detail}")
error.update(
{"status": response.status_code,}
)
return error
class ServiceDocumentDepositClient(BaseDepositClient):
"""Service Document information retrieval.
"""
def __init__(self, config=None, url=None, auth=None):
super().__init__(
url=url,
auth=auth,
config=config,
error_msg="Service document failure at %s: %s",
empty_result={"collection": None},
)
def compute_url(self, *args, **kwargs):
return "/servicedocument/"
def compute_method(self, *args, **kwargs):
return "get"
def parse_result_ok(
self, xml_content: str, headers: Optional[Dict] = None
) -> Dict[str, Any]:
"""Parse service document's success response.
"""
return parse_xml(xml_content)
def parse_result_error(self, xml_content: str) -> Dict[str, Any]:
result = super().parse_result_error(xml_content)
return {"error": result["summary"]}
class StatusDepositClient(BaseDepositClient):
"""Status information on a deposit.
"""
def __init__(self, config=None, url=None, auth=None):
super().__init__(
url=url,
auth=auth,
config=config,
error_msg="Status check failure at %s: %s",
empty_result={
"deposit_status": None,
"deposit_status_detail": None,
"deposit_swh_id": None,
},
)
def compute_url(self, collection, deposit_id):
return "/%s/%s/status/" % (collection, deposit_id)
def compute_method(self, *args, **kwargs):
return "get"
def parse_result_ok(
self, xml_content: str, headers: Optional[Dict] = None
) -> Dict[str, Any]:
"""Given an xml content as string, returns a deposit dict.
"""
data = parse_xml(xml_content)
keys = [
"deposit_id",
"deposit_status",
"deposit_status_detail",
"deposit_swh_id",
"deposit_swh_id_context",
"deposit_external_id",
]
return {key: data.get("swh:" + key) for key in keys}
class CollectionListDepositClient(BaseDepositClient):
"""List a collection of deposits (owned by a user)
"""
def __init__(self, config=None, url=None, auth=None):
super().__init__(
url=url,
auth=auth,
config=config,
error_msg="List deposits failure at %s: %s",
empty_result={},
)
def compute_url(self, collection, **kwargs):
return f"/{collection}/"
def compute_method(self, *args, **kwargs):
return "get"
def compute_params(self, **kwargs) -> Dict[str, Any]:
"""Transmit pagination params if values provided are not None
(e.g. page, page_size)
"""
return {k: v for k, v in kwargs.items() if v is not None}
def parse_result_ok(
self, xml_content: str, headers: Optional[Dict] = None
) -> Dict[str, Any]:
"""Given an xml content as string, returns a deposit dict.
"""
link_header = headers.get("Link", "") if headers else ""
links = parse_header_links(link_header)
data = parse_xml(xml_content)["atom:feed"]
total_result = data.get("swh:count", 0)
keys = [
"id",
"reception_date",
"complete_date",
"external_id",
"swhid",
"status",
"status_detail",
"swhid_context",
"origin_url",
]
entries_ = data.get("atom:entry", [])
entries = [entries_] if isinstance(entries_, dict) else entries_
deposits_d = [
{
key: deposit.get(f"swh:{key}")
for key in keys
if deposit.get(f"swh:{key}") is not None
}
for deposit in entries
]
return {
"count": total_result,
"deposits": deposits_d,
**{entry["rel"]: entry["url"] for entry in links},
}
class BaseCreateDepositClient(BaseDepositClient):
"""Deposit client base class to post new deposit.
"""
def __init__(self, config=None, url=None, auth=None):
super().__init__(
url=url,
auth=auth,
config=config,
error_msg="Post Deposit failure at %s: %s",
empty_result={"swh:deposit_id": None, "swh:deposit_status": None,},
)
def compute_url(self, collection, *args, **kwargs):
return "/%s/" % collection
def compute_method(self, *args, **kwargs):
return "post"
def parse_result_ok(
self, xml_content: str, headers: Optional[Dict] = None
) -> Dict[str, Any]:
"""Given an xml content as string, returns a deposit dict.
"""
data = parse_xml(xml_content)
keys = [
"deposit_id",
"deposit_status",
"deposit_status_detail",
"deposit_date",
]
return {key: data.get("swh:" + key) for key in keys}
def compute_headers(self, info: Dict[str, Any]) -> Dict[str, Any]:
return info
def do_execute(self, method, url, info, **kwargs):
with open(info["filepath"], "rb") as f:
return self.do(method, url, data=f, headers=info["headers"])
class CreateArchiveDepositClient(BaseCreateDepositClient):
"""Post an archive (binary) deposit client."""
def compute_headers(self, info):
headers = {
"CONTENT_MD5": info["md5sum"],
"IN-PROGRESS": str(info["in_progress"]),
"CONTENT-TYPE": info["content-type"],
"CONTENT-DISPOSITION": "attachment; filename=%s" % (info["filename"],),
}
if "slug" in info:
headers["SLUG"] = info["slug"]
return headers
def compute_information(self, *args, **kwargs) -> Dict[str, Any]:
info = compute_unified_information(
*args, filepath=kwargs["archive_path"], **kwargs
)
info["headers"] = self.compute_headers(info)
return info
class UpdateArchiveDepositClient(CreateArchiveDepositClient):
"""Update (add/replace) an archive (binary) deposit client."""
def compute_url(self, collection, *args, deposit_id=None, **kwargs):
return "/%s/%s/media/" % (collection, deposit_id)
def compute_method(self, *args, replace=False, **kwargs):
return "put" if replace else "post"
class CreateMetadataDepositClient(BaseCreateDepositClient):
"""Post a metadata deposit client."""
def compute_headers(self, info):
headers = {
"IN-PROGRESS": str(info["in_progress"]),
"CONTENT-TYPE": "application/atom+xml;type=entry",
}
if "slug" in info:
headers["SLUG"] = info["slug"]
return headers
def compute_information(self, *args, **kwargs) -> Dict[str, Any]:
info = compute_unified_information(
*args, filepath=kwargs["metadata_path"], **kwargs
)
info["headers"] = self.compute_headers(info)
return info
class UpdateMetadataOnPartialDepositClient(CreateMetadataDepositClient):
"""Update (add/replace) metadata on partial deposit scenario."""
def compute_url(self, collection, *args, deposit_id=None, **kwargs):
return f"/{collection}/{deposit_id}/metadata/"
def compute_method(self, *args, replace: bool = False, **kwargs) -> str:
return "put" if replace else "post"
class UpdateMetadataOnDoneDepositClient(CreateMetadataDepositClient):
"""Update metadata on "done" deposit. This requires the deposit swhid."""
def compute_url(self, collection, *args, deposit_id=None, **kwargs):
return f"/{collection}/{deposit_id}/atom/"
def compute_headers(self, info: Dict[str, Any]) -> Dict[str, Any]:
return {
"CONTENT-TYPE": "application/atom+xml;type=entry",
"X_CHECK_SWHID": info["swhid"],
}
def compute_method(self, *args, **kwargs) -> str:
return "put"
class CreateMetadataOnlyDepositClient(BaseCreateDepositClient):
"""Create metadata-only deposit."""
def compute_information(self, *args, **kwargs) -> Dict[str, Any]:
return {
"headers": {"CONTENT-TYPE": "application/atom+xml;type=entry",},
"filepath": kwargs["metadata_path"],
}
def parse_result_ok(
self, xml_content: str, headers: Optional[Dict] = None
) -> Dict[str, Any]:
"""Given an xml content as string, returns a deposit dict.
"""
data = parse_xml(xml_content)
keys = [
"deposit_id",
"deposit_status",
"deposit_date",
]
return {key: data.get("swh:" + key) for key in keys}
class CreateMultipartDepositClient(BaseCreateDepositClient):
"""Create a multipart deposit client."""
def _multipart_info(self, info, info_meta):
files = [
(
"file",
(info["filename"], open(info["filepath"], "rb"), info["content-type"]),
),
(
"atom",
(
info_meta["filename"],
open(info_meta["filepath"], "rb"),
"application/atom+xml",
),
),
]
headers = {
"CONTENT_MD5": info["md5sum"],
"IN-PROGRESS": str(info["in_progress"]),
}
if "slug" in info:
headers["SLUG"] = info["slug"]
return files, headers
def compute_information(self, *args, **kwargs) -> Dict[str, Any]:
info = compute_unified_information(*args, filepath=kwargs["archive_path"],)
info_meta = compute_unified_information(
*args, filepath=kwargs["metadata_path"],
)
files, headers = self._multipart_info(info, info_meta)
return {"files": files, "headers": headers}
def do_execute(self, method, url, info, **kwargs):
return self.do(method, url, files=info["files"], headers=info["headers"])
class UpdateMultipartDepositClient(CreateMultipartDepositClient):
"""Update a multipart deposit client."""
def compute_url(self, collection, *args, deposit_id=None, **kwargs):
return "/%s/%s/metadata/" % (collection, deposit_id)
def compute_method(self, *args, replace=False, **kwargs):
return "put" if replace else "post"
class PublicApiDepositClient(BaseApiDepositClient):
"""Public api deposit client."""
def service_document(self):
"""Retrieve service document endpoint's information."""
return ServiceDocumentDepositClient(url=self.base_url, auth=self.auth).execute()
def deposit_status(self, collection: str, deposit_id: int):
"""Retrieve status information on a deposit."""
return StatusDepositClient(url=self.base_url, auth=self.auth).execute(
collection, deposit_id
)
def deposit_list(
self,
collection: str,
page: Optional[int] = None,
page_size: Optional[int] = None,
):
"""List deposits from the collection"""
return CollectionListDepositClient(url=self.base_url, auth=self.auth).execute(
collection, page=page, page_size=page_size
)
def deposit_create(
self,
collection: str,
slug: Optional[str],
archive: Optional[str] = None,
metadata: Optional[str] = None,
in_progress: bool = False,
):
"""Create a new deposit (archive, metadata, both as multipart)."""
if archive and not metadata:
return CreateArchiveDepositClient(
url=self.base_url, auth=self.auth
).execute(collection, in_progress, slug, archive_path=archive)
elif not archive and metadata:
return CreateMetadataDepositClient(
url=self.base_url, auth=self.auth
).execute(collection, in_progress, slug, metadata_path=metadata)
else:
return CreateMultipartDepositClient(
url=self.base_url, auth=self.auth
).execute(
collection,
in_progress,
slug,
archive_path=archive,
metadata_path=metadata,
)
def deposit_update(
self,
collection: str,
deposit_id: int,
slug: Optional[str],
archive: Optional[str] = None,
metadata: Optional[str] = None,
in_progress: bool = False,
replace: bool = False,
swhid: Optional[str] = None,
):
"""Update (add/replace) existing deposit (archive, metadata, both)."""
response = self.deposit_status(collection, deposit_id)
if "error" in response:
return response
status = response["deposit_status"]
if swhid is None and status != "partial":
return {
"error": "You can only act on deposit with status 'partial'",
"detail": f"The deposit {deposit_id} has status '{status}'",
"deposit_status": status,
"deposit_id": deposit_id,
}
if swhid is not None and status != "done":
return {
"error": "You can only update metadata on deposit with status 'done'",
"detail": f"The deposit {deposit_id} has status '{status}'",
"deposit_status": status,
"deposit_id": deposit_id,
}
if archive and not metadata:
result = UpdateArchiveDepositClient(
url=self.base_url, auth=self.auth
).execute(
collection,
in_progress,
slug,
deposit_id=deposit_id,
archive_path=archive,
replace=replace,
)
elif not archive and metadata and swhid is None:
result = UpdateMetadataOnPartialDepositClient(
url=self.base_url, auth=self.auth
).execute(
collection,
in_progress,
slug,
deposit_id=deposit_id,
metadata_path=metadata,
replace=replace,
)
elif not archive and metadata and swhid is not None:
result = UpdateMetadataOnDoneDepositClient(
url=self.base_url, auth=self.auth
).execute(
collection,
in_progress,
slug,
deposit_id=deposit_id,
metadata_path=metadata,
swhid=swhid,
)
else:
result = UpdateMultipartDepositClient(
url=self.base_url, auth=self.auth
).execute(
collection,
in_progress,
slug,
deposit_id=deposit_id,
archive_path=archive,
metadata_path=metadata,
replace=replace,
)
if "error" in result:
return result
return self.deposit_status(collection, deposit_id)
def deposit_metadata_only(
self, collection: str, metadata: Optional[str] = None,
):
assert metadata is not None
return CreateMetadataOnlyDepositClient(
url=self.base_url, auth=self.auth
).execute(collection, metadata_path=metadata)
diff --git a/swh/deposit/tests/api/test_converters.py b/swh/deposit/tests/api/test_converters.py
index fe16776e..a446d1c7 100644
--- a/swh/deposit/tests/api/test_converters.py
+++ b/swh/deposit/tests/api/test_converters.py
@@ -1,95 +1,98 @@
# Copyright (C) 2017-2019 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
from swh.deposit.api.converters import convert_status_detail
def test_convert_status_detail_empty():
for status_detail in [{}, {"dummy-keys": []}, None]:
assert convert_status_detail(status_detail) is None
def test_convert_status_detail():
status_detail = {
"url": {
"summary": "At least one url field must be compatible with the client's domain name. The following url fields failed the check", # noqa
"fields": ["blahurl", "testurl"],
},
"metadata": [
{"summary": "Mandatory fields missing", "fields": ["url", "title"],},
{
"summary": "Alternate fields missing",
"fields": ["name or title", "url or badurl"],
},
],
"archive": [{"summary": "Unreadable archive", "fields": ["1"],}],
}
expected_status_detail = """- Mandatory fields missing (url, title)
- Alternate fields missing (name or title, url or badurl)
- Unreadable archive (1)
- At least one url field must be compatible with the client's domain name. The following url fields failed the check (blahurl, testurl)
""" # noqa
actual_status_detail = convert_status_detail(status_detail)
assert actual_status_detail == expected_status_detail
def test_convert_status_detail_2():
status_detail = {
"url": {
"summary": "At least one compatible url field. Failed",
"fields": ["testurl"],
},
"metadata": [{"summary": "Mandatory fields missing", "fields": ["name"],},],
"archive": [
{"summary": "Invalid archive", "fields": ["2"],},
{"summary": "Unsupported archive", "fields": ["1"],},
],
+ "loading": ["error1", "error 2"],
}
expected_status_detail = """- Mandatory fields missing (name)
- Invalid archive (2)
- Unsupported archive (1)
- At least one compatible url field. Failed (testurl)
+- error1
+- error 2
"""
actual_status_detail = convert_status_detail(status_detail)
assert actual_status_detail == expected_status_detail
def test_convert_status_detail_3():
status_detail = {
"url": {"summary": "At least one compatible url field",},
}
expected_status_detail = "- At least one compatible url field\n"
actual_status_detail = convert_status_detail(status_detail)
assert actual_status_detail == expected_status_detail
def test_convert_status_detail_edge_case():
status_detail = {
"url": {
"summary": "At least one compatible url field. Failed",
"fields": ["testurl"],
},
"metadata": [
{"summary": "Mandatory fields missing", "fields": ["9", 10, 1.212],},
],
"archive": [
{"summary": "Invalid archive", "fields": ["3"],},
{"summary": "Unsupported archive", "fields": [2],},
],
}
expected_status_detail = """- Mandatory fields missing (9, 10, 1.212)
- Invalid archive (3)
- Unsupported archive (2)
- At least one compatible url field. Failed (testurl)
"""
actual_status_detail = convert_status_detail(status_detail)
assert actual_status_detail == expected_status_detail
diff --git a/swh/deposit/tests/api/test_deposit_private_update_status.py b/swh/deposit/tests/api/test_deposit_private_update_status.py
index c054c4ce..9fd6034e 100644
--- a/swh/deposit/tests/api/test_deposit_private_update_status.py
+++ b/swh/deposit/tests/api/test_deposit_private_update_status.py
@@ -1,195 +1,198 @@
# Copyright (C) 2017-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import copy
import json
from django.urls import reverse_lazy as reverse
from rest_framework import status
from swh.deposit.api.private.deposit_update_status import MANDATORY_KEYS
from swh.deposit.config import (
DEPOSIT_STATUS_LOAD_FAILURE,
DEPOSIT_STATUS_LOAD_SUCCESS,
PRIVATE_PUT_DEPOSIT,
)
from swh.deposit.models import Deposit
PRIVATE_PUT_DEPOSIT_NC = PRIVATE_PUT_DEPOSIT + "-nc"
def private_check_url_endpoints(collection, deposit):
"""There are 2 endpoints to check (one with collection, one without)"""
return [
reverse(PRIVATE_PUT_DEPOSIT, args=[collection.name, deposit.id]),
reverse(PRIVATE_PUT_DEPOSIT_NC, args=[deposit.id]),
]
def test_update_deposit_status_success_with_info(
authenticated_client, deposit_collection, ready_deposit_verified
):
"""Update deposit with load success should require all information to succeed
"""
deposit = ready_deposit_verified
expected_status = DEPOSIT_STATUS_LOAD_SUCCESS
+ status_detail = "it works!"
origin_url = "something"
directory_id = "42a13fc721c8716ff695d0d62fc851d641f3a12b"
revision_id = "47dc6b4636c7f6cba0df83e3d5490bf4334d987e"
snapshot_id = "68c0d26104d47e278dd6be07ed61fafb561d0d20"
full_body_info = {
"status": DEPOSIT_STATUS_LOAD_SUCCESS,
+ "status_detail": status_detail,
"revision_id": revision_id,
"directory_id": directory_id,
"snapshot_id": snapshot_id,
"origin_url": origin_url,
}
for url in private_check_url_endpoints(deposit_collection, deposit):
expected_swhid = "swh:1:dir:%s" % directory_id
expected_swhid_context = (
f"{expected_swhid}"
f";origin={origin_url}"
f";visit=swh:1:snp:{snapshot_id}"
f";anchor=swh:1:rev:{revision_id}"
f";path=/"
)
response = authenticated_client.put(
url, content_type="application/json", data=json.dumps(full_body_info),
)
assert response.status_code == status.HTTP_204_NO_CONTENT
deposit = Deposit.objects.get(pk=deposit.id)
assert deposit.status == expected_status
+ assert deposit.status_detail == status_detail
assert deposit.swhid == expected_swhid
assert deposit.swhid_context == expected_swhid_context
# Reset deposit
deposit = ready_deposit_verified
deposit.save()
def test_update_deposit_status_rejected_with_info(
authenticated_client, deposit_collection, ready_deposit_verified
):
"""Update deposit with rejected status needs few information to succeed
"""
deposit = ready_deposit_verified
for url in private_check_url_endpoints(deposit_collection, deposit):
response = authenticated_client.put(
url,
content_type="application/json",
data=json.dumps({"status": DEPOSIT_STATUS_LOAD_FAILURE}),
)
assert response.status_code == status.HTTP_204_NO_CONTENT
deposit = Deposit.objects.get(pk=deposit.id)
assert deposit.status == DEPOSIT_STATUS_LOAD_FAILURE
assert deposit.swhid is None
assert deposit.swhid_context is None
# Reset status
deposit = ready_deposit_verified
deposit.save()
def test_update_deposit_status_success_with_incomplete_data(
authenticated_client, deposit_collection, ready_deposit_verified
):
"""Update deposit status with status success and incomplete information should fail
"""
deposit = ready_deposit_verified
origin_url = "something"
directory_id = "42a13fc721c8716ff695d0d62fc851d641f3a12b"
revision_id = "47dc6b4636c7f6cba0df83e3d5490bf4334d987e"
snapshot_id = "68c0d26104d47e278dd6be07ed61fafb561d0d20"
new_status = DEPOSIT_STATUS_LOAD_SUCCESS
full_body_info = {
"status": new_status,
"revision_id": revision_id,
"directory_id": directory_id,
"snapshot_id": snapshot_id,
"origin_url": origin_url,
}
for url in private_check_url_endpoints(deposit_collection, deposit):
for key in MANDATORY_KEYS:
# Crafting body with missing information so that it raises
body = copy.deepcopy(full_body_info)
body.pop(key) # make the body incomplete
response = authenticated_client.put(
url, content_type="application/json", data=json.dumps(body),
)
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert (
f"deposit status to {new_status} requires information {key}"
in response.content.decode("utf-8")
)
def test_update_deposit_status_will_fail_with_unknown_status(
authenticated_client, deposit_collection, ready_deposit_verified
):
"""Unknown status for update should return a 400 response
"""
deposit = ready_deposit_verified
for url in private_check_url_endpoints(deposit_collection, deposit):
response = authenticated_client.put(
url, content_type="application/json", data=json.dumps({"status": "unknown"})
)
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert b"Possible status in " in response.content
def test_update_deposit_status_will_fail_with_no_status_key(
authenticated_client, deposit_collection, ready_deposit_verified
):
"""No status provided for update should return a 400 response
"""
deposit = ready_deposit_verified
for url in private_check_url_endpoints(deposit_collection, deposit):
response = authenticated_client.put(
url,
content_type="application/json",
data=json.dumps({"something": "something"}),
)
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert b"The status key is mandatory with possible values" in response.content
def test_update_deposit_status_success_without_swhid_fail(
authenticated_client, deposit_collection, ready_deposit_verified
):
"""Providing successful status without swhid should return a 400
"""
deposit = ready_deposit_verified
for url in private_check_url_endpoints(deposit_collection, deposit):
response = authenticated_client.put(
url,
content_type="application/json",
data=json.dumps({"status": DEPOSIT_STATUS_LOAD_SUCCESS}),
)
assert response.status_code == status.HTTP_400_BAD_REQUEST
assert (
b"Updating deposit status to done requires information" in response.content
)
diff --git a/swh/deposit/tests/loader/test_client.py b/swh/deposit/tests/loader/test_client.py
index 540e0547..8434a0d0 100644
--- a/swh/deposit/tests/loader/test_client.py
+++ b/swh/deposit/tests/loader/test_client.py
@@ -1,242 +1,246 @@
# Copyright (C) 2017-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import json
import os
from typing import Any, Callable, Optional
from urllib.parse import urlparse
import pytest
from requests import Session
from swh.deposit.client import PrivateApiDepositClient
from swh.deposit.config import DEPOSIT_STATUS_LOAD_FAILURE, DEPOSIT_STATUS_LOAD_SUCCESS
CLIENT_TEST_CONFIG = {
"url": "https://nowhere.org/",
"auth": {}, # no authentication in test scenario
}
@pytest.fixture
def deposit_config():
return CLIENT_TEST_CONFIG
def test_client_config(deposit_config_path):
for client in [
# config passed as constructor parameter
PrivateApiDepositClient(config=CLIENT_TEST_CONFIG),
# config loaded from environment
PrivateApiDepositClient(),
]:
assert client.base_url == CLIENT_TEST_CONFIG["url"]
assert client.auth is None
def build_expected_path(datadir, base_url: str, api_url: str) -> str:
"""Build expected path from api to served file
"""
url = urlparse(base_url)
dirname = "%s_%s" % (url.scheme, url.hostname)
if api_url.endswith("/"):
api_url = api_url[:-1]
if api_url.startswith("/"):
api_url = api_url[1:]
suffix_path = api_url.replace("/", "_")
return os.path.join(datadir, dirname, suffix_path)
def test_build_expected_path(datadir):
actual_path = build_expected_path(datadir, "http://example.org", "/hello/you/")
assert actual_path == os.path.join(datadir, "http_example.org", "hello_you")
def read_served_path(
datadir,
base_url: str,
api_url: str,
convert_fn: Optional[Callable[[str], Any]] = None,
) -> bytes:
"""Read served path
"""
archive_path = build_expected_path(datadir, base_url, api_url)
with open(archive_path, "rb") as f:
content = f.read()
if convert_fn:
content = convert_fn(content.decode("utf-8"))
return content
def test_read_served_path(datadir):
actual_content = read_served_path(datadir, "http://example.org", "/hello/you/")
assert actual_content == b"hello people\n"
actual_content2 = read_served_path(
datadir, "http://example.org", "/hello.json", convert_fn=json.loads
)
assert actual_content2 == {"a": [1, 3]}
# private api to retrieve archive
def test_archive_get(tmp_path, datadir, requests_mock_datadir):
"""Retrieving archive data through private api should stream data
"""
api_url = "/1/private/test/1/raw/"
client = PrivateApiDepositClient(CLIENT_TEST_CONFIG)
expected_content = read_served_path(datadir, client.base_url, api_url)
archive_path = os.path.join(tmp_path, "test.archive")
archive_path = client.archive_get(api_url, archive_path)
assert os.path.exists(archive_path) is True
with open(archive_path, "rb") as f:
actual_content = f.read()
assert actual_content == expected_content
assert client.base_url == CLIENT_TEST_CONFIG["url"]
assert client.auth is None
def test_archive_get_auth(tmp_path, datadir, requests_mock_datadir):
"""Retrieving archive data through private api should stream data
"""
api_url = "/1/private/test/1/raw/"
config = CLIENT_TEST_CONFIG.copy()
config["auth"] = { # add authentication setup
"username": "user",
"password": "pass",
}
client = PrivateApiDepositClient(config)
expected_content = read_served_path(datadir, client.base_url, api_url)
archive_path = os.path.join(tmp_path, "test.archive")
archive_path = client.archive_get(api_url, archive_path)
assert os.path.exists(archive_path) is True
with open(archive_path, "rb") as f:
actual_content = f.read()
assert actual_content == expected_content
assert client.base_url == CLIENT_TEST_CONFIG["url"]
assert client.auth == ("user", "pass")
def test_archive_get_ko(tmp_path, datadir, requests_mock_datadir):
"""Reading archive can fail for some reasons
"""
unknown_api_url = "/1/private/unknown/deposit-id/raw/"
client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
with pytest.raises(ValueError, match="Problem when retrieving deposit"):
client.archive_get(unknown_api_url, "some/path")
# private api read metadata
def test_metadata_get(datadir, requests_mock_datadir):
"""Reading archive should write data in temporary directory
"""
api_url = "/1/private/test/1/metadata"
client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
actual_metadata = client.metadata_get(api_url)
assert isinstance(actual_metadata, str) is False
expected_content = read_served_path(
datadir, client.base_url, api_url, convert_fn=json.loads
)
assert actual_metadata == expected_content
def test_metadata_get_ko(requests_mock_datadir):
"""Reading metadata can fail for some reasons
"""
unknown_api_url = "/1/private/unknown/deposit-id/metadata/"
client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
with pytest.raises(ValueError, match="Problem when retrieving metadata"):
client.metadata_get(unknown_api_url)
# private api check
def test_check(requests_mock_datadir):
"""When check ok, this should return the deposit's status
"""
api_url = "/1/private/test/1/check"
client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
r = client.check(api_url)
assert r == "something"
def test_check_fails(requests_mock_datadir):
"""Checking deposit can fail for some reason
"""
unknown_api_url = "/1/private/test/10/check"
client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
with pytest.raises(ValueError, match="Problem when checking deposit"):
client.check(unknown_api_url)
# private api update status
def test_status_update(mocker):
"""Update status
"""
mocked_put = mocker.patch.object(Session, "request")
deposit_client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
deposit_client.status_update(
- "/update/status", DEPOSIT_STATUS_LOAD_SUCCESS, revision_id="some-revision-id",
+ "/update/status",
+ DEPOSIT_STATUS_LOAD_SUCCESS,
+ revision_id="some-revision-id",
+ status_detail="foo bar",
)
mocked_put.assert_called_once_with(
"put",
"https://nowhere.org/update/status",
json={
"status": DEPOSIT_STATUS_LOAD_SUCCESS,
+ "status_detail": "foo bar",
"revision_id": "some-revision-id",
},
)
def test_status_update_with_no_revision_id(mocker):
"""Reading metadata can fail for some reasons
"""
mocked_put = mocker.patch.object(Session, "request")
deposit_client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
deposit_client.status_update("/update/status/fail", DEPOSIT_STATUS_LOAD_FAILURE)
mocked_put.assert_called_once_with(
"put",
"https://nowhere.org/update/status/fail",
json={"status": DEPOSIT_STATUS_LOAD_FAILURE,},
)
diff --git a/swh/deposit/urls.py b/swh/deposit/urls.py
index 1c57bfa1..64aee560 100644
--- a/swh/deposit/urls.py
+++ b/swh/deposit/urls.py
@@ -1,44 +1,44 @@
# Copyright (C) 2017-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
"""SWH's main deposit URL Configuration
"""
from __future__ import annotations
from typing import Sequence, Union
from django.conf.urls import include, url
from django.shortcuts import render
from django.views.generic.base import RedirectView
from rest_framework.urlpatterns import format_suffix_patterns
try:
from django.urls import URLPattern, URLResolver
except ImportError:
# retro-compatibility workaround, django 1.11.29 [1] does not expose the previous
# module, so we fallback to no typing for such version.
# [1] django debian stable version: 1:1.11.29-1~deb10u1
pass
favicon_view = RedirectView.as_view(
url="/static/img/icons/swh-logo-32x32.png", permanent=True
)
-def default_view(req):
+def default_view(req, format=None):
return render(req, "homepage.html")
urlpatterns: Sequence[Union[URLPattern, URLResolver]]
urlpatterns = format_suffix_patterns(
[
url(r"^favicon\.ico$", favicon_view),
url(r"^1/", include("swh.deposit.api.urls")),
url(r"^1/private/", include("swh.deposit.api.private.urls")),
url(r"^$", default_view, name="home"),
]
)

File Metadata

Mime Type
text/x-diff
Expires
Fri, Jul 4, 11:35 AM (3 w, 2 d ago)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
3452542

Event Timeline

About · Developer information · Legal