Page Menu
Home
Software Heritage
Search
Configure Global Search
Log In
Files
F9347295
No One
Temporary
Actions
View File
Edit File
Delete File
View Transforms
Subscribe
Mute Notifications
Award Token
Flag For Later
Size
176 KB
Subscribers
None
View Options
diff --git a/PKG-INFO b/PKG-INFO
index d44b69ca..6127eb6f 100644
--- a/PKG-INFO
+++ b/PKG-INFO
@@ -1,37 +1,37 @@
Metadata-Version: 2.1
Name: swh.deposit
-Version: 0.2.0
+Version: 0.3.0
Summary: Software Heritage Deposit Server
Home-page: https://forge.softwareheritage.org/source/swh-deposit/
Author: Software Heritage developers
Author-email: swh-devel@inria.fr
License: UNKNOWN
Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest
Project-URL: Funding, https://www.softwareheritage.org/donate
Project-URL: Source, https://forge.softwareheritage.org/source/swh-deposit
Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-deposit/
Description: # swh-deposit
This is [Software Heritage](https://www.softwareheritage.org)'s
[SWORD 2.0](http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html) Server
implementation, as well as a simple client to upload deposits on the server.
**S.W.O.R.D** (**S**imple **W**eb-Service **O**ffering **R**epository
**D**eposit) is an interoperability standard for digital file deposit.
This implementation will permit interaction between a client (a
repository) and a server (SWH repository) to permit deposits of
software source code archives and associated metadata.
The documentation is at ./docs/README-specification.md
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 5 - Production/Stable
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Provides-Extra: testing
Provides-Extra: server
diff --git a/conftest.py b/conftest.py
index 16d82778..cda6e0a1 100644
--- a/conftest.py
+++ b/conftest.py
@@ -1,15 +1,19 @@
# Copyright (C) 2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import pytest
-pytest_plugins = ["swh.scheduler.pytest_plugin", "swh.storage.pytest_plugin"]
+pytest_plugins = [
+ "swh.scheduler.pytest_plugin",
+ "swh.storage.pytest_plugin",
+ "swh.core.pytest_plugin",
+]
@pytest.fixture(scope="session")
def swh_scheduler_celery_includes(swh_scheduler_celery_includes):
return swh_scheduler_celery_includes + [
"swh.deposit.loader.tasks",
]
diff --git a/debian/changelog b/debian/changelog
index 4a623ea4..e6a8c878 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,897 +1,906 @@
-swh-deposit (0.2.0-1~swh1~bpo10+1) buster-swh; urgency=medium
+swh-deposit (0.3.0-1~swh1) unstable-swh; urgency=medium
- * Rebuild for buster-swh
-
- -- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 01 Oct 2020 13:31:22 +0000
+ * New upstream release 0.3.0 - (tagged by Antoine R. Dumont
+ (@ardumont) <ardumont@softwareheritage.org> on 2020-10-13 15:12:50
+ +0200)
+ * Upstream changes: - v0.3.0 - deposit_client: Allow deposit
+ metadata update on completed deposit - deposit.client: Improve
+ cli error messages and add missing coverage - cli.client: Add
+ coverage - cli.client: Add types - cli.client: Add types and
+ refactor tests - conftest: Declare swh.core pytest_plugin -
+ deposit: Reuse config.load_from_envvar for configuration loading
+ - test_deposit_content: Add missing coverage
+
+ -- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Tue, 13 Oct 2020 13:16:30 +0000
swh-deposit (0.2.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.2.0 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-10-01 15:25:42
+0200)
* Upstream changes: - v0.2.0 - Allow deposit metadata update
on deposit already completed - Transit raw metadata to the
loader to unify with metadata update scenario - deposit*: Rename
internally swh_id references to swhid - deposit.parsers: Process
namespace when using xmltodict.parse - tests: Add missing update
scenarios - tests: Explicit the bad request scenario error
messages - tests: Ensure all empty body test cases are covered
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 01 Oct 2020 13:29:29 +0000
swh-deposit (0.1.0-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.1.0 - (tagged by David Douard
<david.douard@sdfa3.org> on 2020-09-25 11:49:24 +0200)
* Upstream changes: - v0.1.0
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Fri, 25 Sep 2020 09:53:06 +0000
swh-deposit (0.0.90-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.90 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-06-01 12:25:53
+0200)
* Upstream changes: - v0.0.90 - swh.deposit.models: Upload
deposit archives to dedicated folder
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 01 Jun 2020 10:31:57 +0000
swh-deposit (0.0.89-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.89 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-20 17:45:23
+0200)
* Upstream changes: - v0.0.89 - private/deposit_list: Allow
exclusion patterns from listing
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 20 May 2020 15:49:10 +0000
swh-deposit (0.0.88-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.88 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-20 11:21:31
+0200)
* Upstream changes: - v0.0.88 - Drop swh_anchor_id* columns
from Deposit model
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 20 May 2020 09:28:14 +0000
swh-deposit (0.0.87-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.87 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-15 12:48:31
+0200)
* Upstream changes: - v0.0.87 - Migrate deposit SWHIDs (data)
to the new specification - Update deposit swhid to respect the
latest specification update - Fix typos
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Fri, 15 May 2020 10:52:33 +0000
swh-deposit (0.0.86-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.86 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-11 18:04:03
+0200)
* Upstream changes: - v0.0.86 - origin/master Make deposit
client deal properly with maintenance issues - maintenance
exception: Make the exception appear as raw content
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 11 May 2020 16:07:28 +0000
swh-deposit (0.0.85-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.85 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-11 15:29:11
+0200)
* Upstream changes: - v0.0.85 - origin/master Align 503
exceptions output format with existing errors - Add deposit
exception handler to improve default error display - tox: Drop
django1 entry, we use django2 in production - settings.common:
Format
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 11 May 2020 13:40:23 +0000
swh-deposit (0.0.84-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.84 - (tagged by Antoine R. Dumont
(@ardumont) <ardumont@softwareheritage.org> on 2020-05-07 14:48:54
+0100)
* Upstream changes: - v0.0.84 - test: Add checks scenario test
cases - test_task: Mark some test to be ignored during debian
build
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 07 May 2020 13:52:51 +0000
swh-deposit (0.0.83-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.83 - (tagged by Valentin Lorentz
<vlorentz@softwareheritage.org> on 2020-05-07 11:52:28 +0200)
* Upstream changes: - v0.0.83 - * Pass collection + id to the
checker instead of an URL.
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 07 May 2020 09:56:40 +0000
swh-deposit (0.0.82-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.82 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2020-04-23 16:40:34
+0200)
* Upstream changes: - v0.0.82 - deposit_read: Simplify api to
return only relevant deposit information - setup: Update the
minimum required runtime python3 version - spec: reference SWHID
using explicit anchors - Update type annotations and signatures
to match djangorestframework-stubs - pytest.ini: Avoid loading
flask plugin to prevent fixture name clash - Add a
pyproject.toml file to target py37 for black - Enable black -
tests: Adapt init_sentry api change call with environment parameter
- docs: Fix sphinx warnings - tests/gunicorn_config: Fix tests
after recent changes in swh-core
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 23 Apr 2020 14:45:49 +0000
swh-deposit (0.0.81-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.81 - (tagged by David Douard
<david.douard@sdfa3.org> on 2020-01-13 11:50:07 +0100)
* Upstream changes: - v0.0.81
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 13 Jan 2020 11:07:54 +0000
swh-deposit (0.0.80-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.80 - (tagged by David Douard
<david.douard@sdfa3.org> on 2020-01-09 15:54:57 +0100)
* Upstream changes: - v0.0.80
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 09 Jan 2020 15:00:32 +0000
swh-deposit (0.0.79-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.79 - (tagged by Valentin Lorentz
<vlorentz@softwareheritage.org> on 2019-12-19 17:37:41 +0100)
* Upstream changes: - v0.0.79 - * requirements: Pin mypy and
django-stubs version - * homepage: Improve sentence phrasing
- * deposit.api: Add a basic api page to avoid broken link - *
removed dead and deprecated code - * Add sentry integration.
- * Fix log level + status code of the client CLI in case of error.
- * Improve validation of --author and --name. - * Update
documentation of --author to use names instead of emails.
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 19 Dec 2019 16:42:28 +0000
swh-deposit (0.0.78-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.78 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-11-25 18:28:13
+0100)
* Upstream changes: - v0.0.78 - deposit.signal: Simplify
configuration entry
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 25 Nov 2019 17:34:33 +0000
swh-deposit (0.0.77-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.77 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-11-25 14:48:59
+0100)
* Upstream changes: - v0.0.77 - deposit.signals: Send
versioned scheduler tasks - deposit.signals: Scheduler load-
deposit task with new endpoints - mypy: Fix missing import
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 25 Nov 2019 14:02:38 +0000
swh-deposit (0.0.76-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.76 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-11-25 12:26:07
+0100)
* Upstream changes: - v0.0.76 - Start adding mypy annotation
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Mon, 25 Nov 2019 11:29:48 +0000
swh-deposit (0.0.75-1~swh2) unstable-swh; urgency=medium
* Add egg-info to pybuild.testfiles
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 20 Nov 2019 15:06:31 +0100
swh-deposit (0.0.75-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.75 - (tagged by Nicolas Dandrimont
<nicolas@dandrimont.eu> on 2019-10-30 16:50:25 +0100)
* Upstream changes: - Release swh.deposit v0.0.75 - Revert
changes to the task signature until the new loader is ready -
Migrate tests to pytest
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 30 Oct 2019 15:54:55 +0000
swh-deposit (0.0.74-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.74 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-10-09 11:30:00
+0200)
* Upstream changes: - v0.0.74 - deposit.signals: Scheduler
load-deposit task with new endpoints - deposit.private.api:
Expose new endpoints with no collection name - models: Migrate
model to enforce check on delete - setup: register the worker
task in the swh.workers entrypoint - tests: Explicit private
tests in their names - admin CLI: avoid redefining deposit name
in admin subcommand
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 09 Oct 2019 09:35:03 +0000
swh-deposit (0.0.73-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.73 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-09-05 10:15:29
+0200)
* Upstream changes: - v0.0.73 - loader: Add missing visit_type
attribute. - cli/client: Simplify url definition to use -
cli/admin: Add the default domain value to empty - deposit_data:
Remove no longer used DepositRequestType references - doc/sys-
info: Clarify commands - docs: add code of conduct document
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 05 Sep 2019 08:19:17 +0000
swh-deposit (0.0.72-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.72 - (tagged by Valentin Lorentz
<vlorentz@softwareheritage.org> on 2019-06-18 17:15:53 +0200)
* Upstream changes: - Remove argument origin_id from call to
Loader.send_origin_metadata. - It no longer needs that argument.
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 19 Jun 2019 14:26:44 +0000
swh-deposit (0.0.71-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.71 - (tagged by Antoine Lambert
<antoine.lambert@inria.fr> on 2019-05-23 11:02:05 +0200)
* Upstream changes: - version 0.0.71
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 23 May 2019 09:09:54 +0000
swh-deposit (0.0.70-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.70 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-05-10 12:04:33
+0200)
* Upstream changes: - v0.0.70 - Update documentations (getting-
started, metadata) - Improve cli client (expose new status
subcommand, clarify help messages) - Fixes some issues
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Fri, 10 May 2019 10:33:19 +0000
swh-deposit (0.0.69-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.69 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-05-09 10:25:43
+0200)
* Upstream changes: - v0.0.69 - cli.admin: Add an admin
subcommand 'deposit reschedule' - models: Keep scheduler task
ids reference on deposit model - cli: make the deposit cli
command a subcommand of the main 'swh' one - docs: update the
getting-started document to use 'swh deposit upload' command -
docs: Update the sys-info document to expose the `swh deposit admin
deposit - reschedule` subcommand
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 09 May 2019 08:35:53 +0000
swh-deposit (0.0.68-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.68 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-04-17 15:39:52
+0200)
* Upstream changes: - v0.0.68
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 17 Apr 2019 13:50:21 +0000
swh-deposit (0.0.67-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.67 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-02-20 17:39:59
+0100)
* Upstream changes: - v0.0.67 - settings.prod: Use
SWH_CONFIG_FILENAME to load & check swh config - requirements-
swh.txt: Update minimal version required for loader-tar - remove
debian/ tree from master branch
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Wed, 20 Feb 2019 16:45:22 +0000
swh-deposit (0.0.66-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.66 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-02-16 10:22:30
+0100)
* Upstream changes: - v0.0.66 - deposit.loader.checker: Fix
logger initialization
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Sat, 16 Feb 2019 09:27:54 +0000
swh-deposit (0.0.65-1~swh1) unstable-swh; urgency=medium
* New upstream release 0.0.65 - (tagged by Antoine R. Dumont
(@ardumont) <antoine.romain.dumont@gmail.com> on 2019-02-14 18:28:48
+0100)
* Upstream changes: - v0.0.65 - swh/manage: Fix flake8 warning
- Bump dependency on swh-scheduler 0.0.39 - Rewrite celery tasks
as a decorated function - loader.scheduler: Remove non
production code - deposit.loader.tasks: Add tests
-- Software Heritage autobuilder (on jenkins-debian1) <jenkins@jenkins-debian1.internal.softwareheritage.org> Thu, 14 Feb 2019 17:34:09 +0000
swh-deposit (0.0.63-1~swh2) unstable-swh; urgency=medium
* New upstream release, fixing dependencies
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 14 Feb 2019 18:16:04 +0100
swh-deposit (0.0.63-1~swh1) unstable-swh; urgency=medium
* v0.0.63
* deposit_list: Return status_detail as string message and not as
nested
* dict
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 17 Sep 2018 16:23:58 +0200
swh-deposit (0.0.62-1~swh1) unstable-swh; urgency=medium
* v0.0.62
* private.deposit_list: Make the endpoint private
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 12 Sep 2018 15:58:21 +0200
swh-deposit (0.0.61-1~swh1) unstable-swh; urgency=medium
* v0.0.61
* Add api endpoint to list deposits with pagination
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 12 Sep 2018 14:44:58 +0200
swh-deposit (0.0.60-1~swh1) unstable-swh; urgency=medium
* v0.0.60
* Fix production issue regarding deposit status endpoint.
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 04 Sep 2018 18:30:01 +0200
swh-deposit (0.0.59-1~swh1) unstable-swh; urgency=medium
* v0.0.59
* deposit.utils: Fix the potential metadata information loss
* docs: Add sparse/metadata deposit specs
* docs: Update documentation about persistent id with context
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 24 Jul 2018 14:15:55 +0200
swh-deposit (0.0.58-1~swh1) unstable-swh; urgency=medium
* v0.0.58
* d/*: Update to latest python3-swh.model dependency version
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 23 Jul 2018 14:31:46 +0200
swh-deposit (0.0.57-1~swh1) unstable-swh; urgency=medium
* v0.0.57
* swh.deposit.client: Simplify client parsing
* api/deposit_status: Make swh-id be a directory id derivative
* swh.deposit.models: Keep deposit request's raw metadata
* bin: Migrate internal script to use the deposit client
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 23 Jul 2018 13:43:23 +0200
swh-deposit (0.0.56-1~swh1) unstable-swh; urgency=medium
* v0.0.56
* docs: Update deposit with status rejected documentation
* deposit_status: Update the deposit status endpoint with details for
* rejected deposit
* deposit_check: Reject invalid deposit (associated archive containing
* only single archive)
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 17 Jul 2018 13:27:35 +0200
swh-deposit (0.0.55-1~swh1) unstable-swh; urgency=medium
* v0.0.55
* deposit_status: Open detailed status when a deposit fails the checks
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 11 Jul 2018 12:06:10 +0200
swh-deposit (0.0.54-1~swh1) unstable-swh; urgency=medium
* v0.0.54
* deposit_check: Improve details in failing checks
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 10 Jul 2018 10:23:35 +0200
swh-deposit (0.0.53-1~swh1) unstable-swh; urgency=medium
* v0.0.53
* swh.deposit.parsers: Fix xml parsing to not lose duplicated entries
* swh.deposit.tests: Make sure failing tests are complete
* deposit_update: Fix check error during update with wrong mimetype
* deposit_read: Persistent identifier representation has changed
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 06 Jul 2018 16:26:08 +0200
swh-deposit (0.0.52-1~swh1) unstable-swh; urgency=medium
* v0.0.52
* Make the deposit's scheduler configuration adjustable
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 03 May 2018 15:16:30 +0200
swh-deposit (0.0.51-1~swh1) unstable-swh; urgency=medium
* v0.0.51
* Improve origin_visit initialization step
* Properly sandbox the prepare statement so that if it breaks, we can
* update appropriately the visit with the correct status
* Let the visit date be set in lower layer
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 07 Mar 2018 11:07:11 +0100
swh-deposit (0.0.50-1~swh1) unstable-swh; urgency=medium
* v0.0.50
* Bump requirements up for new swh.loader.tar
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 12 Feb 2018 11:21:03 +0100
swh-deposit (0.0.49-1~swh1) unstable-swh; urgency=medium
* Release swh.deposit v0.0.49
* Use snapshots instead of occurrences
-- Nicolas Dandrimont <nicolas@dandrimont.eu> Tue, 06 Feb 2018 14:48:53 +0100
swh-deposit (0.0.48-1~swh1) unstable-swh; urgency=medium
* v0.0.48
* swh.deposit.api.private: Fix revision message missing client name
* docs: simplify endpoints and delete all sword text
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 02 Feb 2018 08:37:39 +0100
swh-deposit (0.0.47-1~swh1) unstable-swh; urgency=medium
* v0.0.47
* swh.loader: Be consistent in returning loader result in task
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 30 Jan 2018 19:03:49 +0100
swh-deposit (0.0.46-1~swh1) unstable-swh; urgency=medium
* v0.0.46
* swh.deposit.client: Explicit private api client
* swh.deposit.client.cli: Fix flag compatibility issue
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 30 Jan 2018 12:19:09 +0100
swh-deposit (0.0.45-1~swh1) unstable-swh; urgency=medium
* v0.0.45
* Simplify collection name retrieval
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 29 Jan 2018 18:09:29 +0100
swh-deposit (0.0.44-1~swh1) unstable-swh; urgency=medium
* v0.0.44
* docs: Update getting-started documentations
* swh.deposit.api: Add swh_id key to None by default in status
endpoint
* swh.deposit.api: Fix tar archive permission in update endpoints
* swh.deposit.api.private: Fix pep8 violation about catch Exception
* swh.deposit.api: Do not hardcode the server uri in service document
* endpoint
* swh.deposit.client: Add a deposit client
* d/control: Create python3-swh.deposit.client package
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 29 Jan 2018 17:41:12 +0100
swh-deposit (0.0.43-1~swh1) unstable-swh; urgency=medium
* v0.0.43
* swh.deposit.api: Deposit returns persistent identifiers
* swh.deposit.api: Rename deposit statuses
* swh.deposit: Support standard tarball formats (.tar.*)
* docs: Update documentation about new archive format support
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 17 Jan 2018 12:02:20 +0100
swh-deposit (0.0.42-1~swh1) unstable-swh; urgency=medium
* v0.0.42
* Fix cosmetic issues in deposit.s.o
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 10 Jan 2018 16:25:48 +0100
swh-deposit (0.0.41-1~swh1) unstable-swh; urgency=medium
* v0.0.41
* swh.deposit.checks: Add url validation
* swh.deposit: Add splash screen to homepage
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 10 Jan 2018 12:12:43 +0100
swh-deposit (0.0.40-1~swh1) unstable-swh; urgency=medium
* v0.0.40
* Fix corner case on deposit checks and clarify intents on checks
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 08 Jan 2018 18:21:38 +0100
swh-deposit (0.0.39-1~swh1) unstable-swh; urgency=medium
* v0.0.39
* Amend detail status message
* docs: Fix url
* Fix check of deposit without content scenario
* Refactor metadata check with pythonic function
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 08 Jan 2018 12:43:15 +0100
swh-deposit (0.0.38-1~swh1) unstable-swh; urgency=medium
* v0.0.38
* Provide existing swh id in the status api when it exists
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 08 Dec 2017 09:41:21 +0100
swh-deposit (0.0.37-1~swh1) unstable-swh; urgency=medium
* v0.0.37
* Adapt to latest dependency on loader-core and storage
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 07 Dec 2017 15:30:32 +0100
swh-deposit (0.0.36-1~swh1) unstable-swh; urgency=medium
* v0.0.36
* swh.deposit.api.private: Deposit's author is swh
* d/control: Bump to latest dependencies version
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 06 Dec 2017 12:21:45 +0100
swh-deposit (0.0.35-1~swh1) unstable-swh; urgency=medium
* v0.0.35
* swh.deposit.loader: Fix intermediary status
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 05 Dec 2017 19:23:40 +0100
swh-deposit (0.0.34-1~swh1) unstable-swh; urgency=medium
* v0.0.34
* d/control: Bump to latest version
* Fix client config typo for local url
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 05 Dec 2017 15:43:59 +0100
swh-deposit (0.0.33-1~swh1) unstable-swh; urgency=medium
* v0.0.33
* Bump to latest swh.loader.tar
* dev swh.deposit.api: Add parent deposit to deposit at creation time
* swh.deposit.service: Clean dead code
* bin/Makefile: Add multipart deposit sample script
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 04 Dec 2017 18:59:50 +0100
swh-deposit (0.0.32-1~swh1) unstable-swh; urgency=medium
* v0.0.32
* Migrate swh.deposit.injection module to swh.deposit.loader
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 30 Nov 2017 16:27:19 +0100
swh-deposit (0.0.31-1~swh1) unstable-swh; urgency=medium
* v0.0.31
* swh.deposit.api: Separate public/private api
* swh.deposit.api.common: Fix authentication issue in production
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 30 Nov 2017 13:18:15 +0100
swh-deposit (0.0.30-1~swh1) unstable-swh; urgency=medium
* v0.0.30
* swh.deposit.signals: Fix wrong task scheduling argument
* swh.deposit.migration: Remove default value
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 29 Nov 2017 18:34:29 +0100
swh-deposit (0.0.29-1~swh1) unstable-swh; urgency=medium
* v0.0.29
* Fix inconsistent term in code
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 29 Nov 2017 15:22:36 +0100
swh-deposit (0.0.28-1~swh1) unstable-swh; urgency=medium
* v0.0.28
* swh.deposit: Use revision_id as swh_id
* swh.deposit: Untangle checks from the current client/server requests
flow
* swh.deposit.injection: Trigger scheduling of checks on deposit
* swh.deposit.injection: Trigger scheduling of loading on deposit
* swh.deposit.injection: Add origin_metadata during loading
* d/control: Bump to latest swh layers
* swh.deposit.tests: Add and refactor tests
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 29 Nov 2017 15:03:56 +0100
swh-deposit (0.0.27-1~swh1) unstable-swh; urgency=medium
* v0.0.27
* swh.deposit.parsers: Fix edge case about decimal serialization
* swh.deposit.injection.loader: Add test on loading a deposit
* swh.loader.test: Fix path initialization
* swh.deposit.injection.scheduler: Adapt default task
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 17 Nov 2017 10:54:08 +0100
swh-deposit (0.0.26-1~swh1) unstable-swh; urgency=medium
* v0.0.26
* swh.deposit: Be consistent in the deposit_status key returned
* docs: Move actual docs inside the docs/ folder
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 25 Oct 2017 17:37:02 +0200
swh-deposit (0.0.25-1~swh1) unstable-swh; urgency=medium
* v0.0.25
* swh.deposit.production: Add support for proxy headers in django
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 24 Oct 2017 14:08:42 +0200
swh-deposit (0.0.24-1~swh1) unstable-swh; urgency=medium
* v0.0.24
* swh.deposit.api: Fix 500 error when browsing api through browser
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 23 Oct 2017 15:37:11 +0200
swh-deposit (0.0.23-1~swh1) unstable-swh; urgency=medium
* v0.0.23
* swh.deposit.api: Deal with new 'rejected' status on deposit
* docs: Update documentation to latest development
* swh.deposit.api: Update docstrings properly
* swh.deposit.api: Add post check validation on deposit
* swh.deposit.tests: Add edge case scenario tests (upload size limit,
* deposit read archives, etc...)
* swh.deposit.api: Fix mismatch hash check message
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 23 Oct 2017 11:31:39 +0200
swh-deposit (0.0.22-1~swh1) unstable-swh; urgency=medium
* v0.0.22
* swh.deposit.api: Return fqdn urls
* swh.deposit.api: Use variable to define the pivot status 'ready'
* swh.deposit.api: Add state iri in the deposit receipt
* swh.deposit.api: Add deposit's status in the deposit receipt
* swh.deposit.api: Explicit no support for Metadata-Relevant header
* swh.deposit.tests: Fix potential listing error
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 19 Oct 2017 15:38:01 +0200
swh-deposit (0.0.21-1~swh1) unstable-swh; urgency=medium
* v0.0.21
* swh.deposit.api: Simplify clean up temporary directory routine for
* deposit read archive api
* swh.deposit.tests: Add missing test cases around deposit read api
* README-injection: Improve sentence phrasing
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 18 Oct 2017 11:36:40 +0200
swh-deposit (0.0.20-1~swh1) unstable-swh; urgency=medium
* v0.0.20
* swh.deposit.scheduler: Move scheduling part to swh.deposit.injection
* swh.deposit.injection: Separation of concern between reading/loading
* swh.deposit.api: Remove dead code
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 16 Oct 2017 18:23:19 +0200
swh-deposit (0.0.19-1~swh1) unstable-swh; urgency=medium
* v0.0.19
* Define summary message for method not allowed endpoints
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 16 Oct 2017 12:46:11 +0200
swh-deposit (0.0.18-1~swh1) unstable-swh; urgency=medium
* v0.0.18
* swh.deposit.api.deposit: make slug header mandatory
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Sat, 14 Oct 2017 10:45:08 +0200
swh-deposit (0.0.17-1~swh1) unstable-swh; urgency=medium
* v0.0.17
* Fix missing files for packaging
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 13 Oct 2017 16:37:27 +0200
swh-deposit (0.0.16-1~swh1) unstable-swh; urgency=medium
* v0.0.16
* packaging: Split python3-swh.deposit / python3-swh.deposit.injection
* swh.deposit.injection: Add deposit archive ingestion task
* packaging: Cleanup
* swh.deposit.auth: Cleanup authentication to use directly drf's
* swh.deposit.api: Split between private and public api
* swh.deposit.api: Add private api to update deposit's status
* swh.deposit.scheduler.cli: Add one-shot task scheduling machinery
* swh.deposit.tests: use the collection name when creating uri
* swh.deposit.admin: Clean up unused code
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 13 Oct 2017 15:41:47 +0200
swh-deposit (0.0.15-1~swh1) unstable-swh; urgency=medium
* v0.0.15
* swh.deposit.api: Add service to clean up temporary archives
* swh.deposit.api: Add private api to read a deposit's raw content
* swh.deposit.api: Update docstring
* swh.deposit.api: Switch from objstorage layer to django's
* docs: Fix typo in private yaml sample
* README-dev: Update documentation about bootstraping the dev env
* swh.deposit.tests: Check existence before directory cleanup
* Remove reference to noop and verbose since no longer in spec 2.0
* bin: Reference sample executables to exercise local run
* swh.deposit.scheduler.cli: Add a scheduling implementation on
deposit
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 09 Oct 2017 10:23:48 +0200
swh-deposit (0.0.14-1~swh1) unstable-swh; urgency=medium
* v0.0.14
* swh.deposit.tests: Add missing test cases scenario about updates
* docs: Improve and make the documentation browsable through browser
* docs: Add README-sys, README-getting-started, README-injection
* swh.deposit.create_user: Fix collection setup
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 28 Sep 2017 16:56:54 +0200
swh-deposit (0.0.13-1~swh1) unstable-swh; urgency=medium
* v0.0.13
* swh.deposit.api: Restrict access to one's own collection
* swh.deposit.api: Unify checks on all endpoints
* swh.deposit: Separate the collection from the client notion
* swh.deposit.api: Add delete deposit endpoint
* swh.deposit.api: Add delete content (archives) from deposit
* swh.deposit.api: Empty post on EDIT-IRI can finalize a deposit
* swh.deposit.model: Relax unicity constraint on external id
* swh.deposit.api: PUT does not permit to have the deposit_id None
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 25 Sep 2017 18:02:46 +0200
swh-deposit (0.0.12-1~swh1) unstable-swh; urgency=medium
* v0.0.12
* swh.deposit.api: Separate the replace metadata from the replace
* archive routine
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 22 Sep 2017 18:59:48 +0200
swh-deposit (0.0.11-1~swh1) unstable-swh; urgency=medium
* v0.0.11
* swh.deposit.static: Add static folder
* swh.deposit.api: Accept modifications to deposit only in partial
status
* swh.deposit.api: Update/Add new deposit metadata/archive
* swh.deposit.api.tests: Tests new use cases
* README: Update and simplify documentation
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Fri, 22 Sep 2017 16:24:25 +0200
swh-deposit (0.0.10-1~swh1) unstable-swh; urgency=medium
* v0.0.10
* swh.deposit.config: Centralize default config in .config module
* swh.deposit.api: Split api module definition
* swh.deposit.auth: Improve white-listing mechanism
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Thu, 21 Sep 2017 10:41:04 +0200
swh-deposit (0.0.9-1~swh1) unstable-swh; urgency=medium
* v0.0.9
* swh.deposit.api: white list / from authentication
* swh.deposit.api: Update state iri endpoint
* swh.deposit.api: Update new IRI endpoints to deal with update
* swh.deposit.api.deposit: Clean up dead code
* README: Update specification on IRIs
* swh.deposit.urls: Fix endpoints to finish with trailing /
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 20 Sep 2017 18:12:33 +0200
swh-deposit (0.0.8-1~swh1) unstable-swh; urgency=medium
* v0.0.8
* swh.deposit.settings: Split logging configuration per platform
* swh.deposit.settings.common: Prefer configuration over code
* swh.deposit.api: Enforce basic authentication
* swh.deposit: Clarify SWHDefaultConfig class's intent
* clean up: Removing user test api endpoints
* doc: Update docstrings
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 19 Sep 2017 14:23:05 +0200
swh-deposit (0.0.7-1~swh1) unstable-swh; urgency=medium
* v0.0.7
* Add swh.deposit.create_user routine to setup user information
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 19 Sep 2017 10:44:59 +0200
swh-deposit (0.0.6-1~swh1) unstable-swh; urgency=medium
* v0.0.6
* Make the packages include all that's needed (templates, fixtures)
* Fix typo in production settings file
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 18 Sep 2017 16:17:53 +0200
swh-deposit (0.0.5-1~swh1) unstable-swh; urgency=medium
* v0.0.5
* Package and add missing python3-djangorestframework-xml
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Mon, 18 Sep 2017 15:10:01 +0200
swh-deposit (0.0.4-1~swh1) unstable-swh; urgency=medium
* v0.0.4
* README: Update spec documentation according to latest development
* README-dev: Initiate a development readme to explicit the local
* dev/production mode
* swh.deposit.settings: Split profile configuration per deployment
* platform (dev, production)
* swh.deposit.views:
* Adapt returned errors to be sword compliant
* Change starting api route endpoints
* Improve deposit request headers checks
* swh.deposit.tests:
* Inhibit side-effects in tests (objstorage, configuration loading,
etc...)
* Add authentication in tests
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Sat, 16 Sep 2017 15:38:44 +0200
swh-deposit (0.0.3-1~swh1) unstable-swh; urgency=medium
* v0.0.3
* Migrate to django framework
* Deployment tryouts
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Tue, 01 Aug 2017 12:36:22 +0200
swh-deposit (0.0.2-1~swh1) unstable-swh; urgency=medium
* v0.0.2
* Fix db connection initialization
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 14 Jun 2017 13:49:31 +0200
swh-deposit (0.0.1-1~swh1) unstable-swh; urgency=medium
* Initial release
* v0.0.1
* Add basic server implementation for deployment testing
-- Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com> Wed, 14 Jun 2017 12:48:37 +0200
diff --git a/pytest.ini b/pytest.ini
index 020ea949..5feb18df 100644
--- a/pytest.ini
+++ b/pytest.ini
@@ -1,9 +1,9 @@
[pytest]
# Remove the pytest_swh_* entries when they stop getting imported automatically
-addopts = -p no:flask -p no:pytest_swh_scheduler -p no:pytest_swh_storage
+addopts = -p no:flask -p no:pytest_swh_scheduler -p no:pytest_swh_storage -p no:pytest_swh_core
norecursedirs = docs .*
DJANGO_SETTINGS_MODULE = swh.deposit.settings.testing
markers =
db: execute tests using a postgresql database
fs: execute tests using the filesystem
diff --git a/requirements-swh-server.txt b/requirements-swh-server.txt
index 5e81fabe..a54273e7 100644
--- a/requirements-swh-server.txt
+++ b/requirements-swh-server.txt
@@ -1,4 +1,4 @@
-swh.core[http]
+swh.core[http] >= 0.4
swh.loader.core >= 0.0.71
swh.scheduler >= 0.0.39
swh.model >= 0.3.8
diff --git a/requirements-swh.txt b/requirements-swh.txt
index 9bc67248..d6d8f166 100644
--- a/requirements-swh.txt
+++ b/requirements-swh.txt
@@ -1 +1 @@
-swh.core[http] >= 0.3
+swh.core[http] >= 0.4
diff --git a/swh.deposit.egg-info/PKG-INFO b/swh.deposit.egg-info/PKG-INFO
index d44b69ca..6127eb6f 100644
--- a/swh.deposit.egg-info/PKG-INFO
+++ b/swh.deposit.egg-info/PKG-INFO
@@ -1,37 +1,37 @@
Metadata-Version: 2.1
Name: swh.deposit
-Version: 0.2.0
+Version: 0.3.0
Summary: Software Heritage Deposit Server
Home-page: https://forge.softwareheritage.org/source/swh-deposit/
Author: Software Heritage developers
Author-email: swh-devel@inria.fr
License: UNKNOWN
Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest
Project-URL: Funding, https://www.softwareheritage.org/donate
Project-URL: Source, https://forge.softwareheritage.org/source/swh-deposit
Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-deposit/
Description: # swh-deposit
This is [Software Heritage](https://www.softwareheritage.org)'s
[SWORD 2.0](http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html) Server
implementation, as well as a simple client to upload deposits on the server.
**S.W.O.R.D** (**S**imple **W**eb-Service **O**ffering **R**epository
**D**eposit) is an interoperability standard for digital file deposit.
This implementation will permit interaction between a client (a
repository) and a server (SWH repository) to permit deposits of
software source code archives and associated metadata.
The documentation is at ./docs/README-specification.md
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 5 - Production/Stable
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Provides-Extra: testing
Provides-Extra: server
diff --git a/swh.deposit.egg-info/SOURCES.txt b/swh.deposit.egg-info/SOURCES.txt
index 7a6f24c0..0e0c6243 100644
--- a/swh.deposit.egg-info/SOURCES.txt
+++ b/swh.deposit.egg-info/SOURCES.txt
@@ -1,220 +1,234 @@
.gitignore
.pre-commit-config.yaml
AUTHORS
CODE_OF_CONDUCT.md
CONTRIBUTORS
LICENSE
MANIFEST.in
Makefile
Makefile.local
README.md
conftest.py
mypy.ini
pyproject.toml
pytest.ini
requirements-server.txt
requirements-swh-server.txt
requirements-swh.txt
requirements-test.txt
requirements.txt
setup.cfg
setup.py
tox.ini
bin/Makefile
bin/content.sh
bin/create_deposit.sh
bin/create_deposit_atom.sh
bin/create_deposit_with_metadata.sh
bin/default-setup
bin/download-deposit-archive.sh
bin/home.sh
bin/replace-deposit-archive.sh
bin/service-document.sh
bin/status.sh
bin/update-deposit-with-another-archive.sh
bin/update-status.sh
docs/.gitignore
docs/Makefile
docs/conf.py
docs/dev-info.rst
docs/getting-started.rst
docs/index.rst
docs/metadata.rst
docs/spec-api.rst
docs/sys-info.rst
docs/_static/.placeholder
docs/_templates/.placeholder
docs/endpoints/collection.rst
docs/endpoints/content.rst
docs/endpoints/service-document.rst
docs/endpoints/status.rst
docs/endpoints/update-media.rst
docs/endpoints/update-metadata.rst
docs/images/deposit-create-chart.png
docs/images/deposit-delete-chart.png
docs/images/deposit-update-chart.png
docs/images/status.png
docs/specs/blueprint.rst
docs/specs/metadata_example.xml
docs/specs/spec-loading.rst
docs/specs/spec-meta-deposit.rst
docs/specs/spec-sparse-deposit.rst
docs/specs/spec-technical.rst
docs/specs/specs.rst
docs/specs/swh.xsd
docs/tests/tests_HAL.rst
resources/deposit/server.yml
swh/__init__.py
swh.deposit.egg-info/PKG-INFO
swh.deposit.egg-info/SOURCES.txt
swh.deposit.egg-info/dependency_links.txt
swh.deposit.egg-info/entry_points.txt
swh.deposit.egg-info/requires.txt
swh.deposit.egg-info/top_level.txt
swh/deposit/__init__.py
swh/deposit/apps.py
swh/deposit/auth.py
swh/deposit/client.py
swh/deposit/config.py
swh/deposit/errors.py
swh/deposit/exception.py
swh/deposit/gunicorn_config.py
swh/deposit/manage.py
swh/deposit/models.py
swh/deposit/parsers.py
swh/deposit/py.typed
swh/deposit/urls.py
swh/deposit/utils.py
swh/deposit/api/__init__.py
swh/deposit/api/checks.py
swh/deposit/api/common.py
swh/deposit/api/converters.py
swh/deposit/api/deposit.py
swh/deposit/api/deposit_content.py
swh/deposit/api/deposit_status.py
swh/deposit/api/deposit_update.py
swh/deposit/api/service_document.py
swh/deposit/api/urls.py
swh/deposit/api/private/__init__.py
swh/deposit/api/private/deposit_check.py
swh/deposit/api/private/deposit_list.py
swh/deposit/api/private/deposit_read.py
swh/deposit/api/private/deposit_update_status.py
swh/deposit/api/private/urls.py
swh/deposit/cli/__init__.py
swh/deposit/cli/admin.py
swh/deposit/cli/client.py
swh/deposit/fixtures/__init__.py
swh/deposit/fixtures/deposit_data.yaml
swh/deposit/loader/__init__.py
swh/deposit/loader/checker.py
swh/deposit/loader/tasks.py
swh/deposit/migrations/0001_initial.py
swh/deposit/migrations/0002_depositrequest_archive.py
swh/deposit/migrations/0003_temporaryarchive.py
swh/deposit/migrations/0004_delete_temporaryarchive.py
swh/deposit/migrations/0005_auto_20171019_1436.py
swh/deposit/migrations/0006_depositclient_url.py
swh/deposit/migrations/0007_auto_20171129_1609.py
swh/deposit/migrations/0008_auto_20171130_1513.py
swh/deposit/migrations/0009_deposit_parent.py
swh/deposit/migrations/0010_auto_20180110_0953.py
swh/deposit/migrations/0011_auto_20180115_1510.py
swh/deposit/migrations/0012_deposit_status_detail.py
swh/deposit/migrations/0013_depositrequest_raw_metadata.py
swh/deposit/migrations/0014_auto_20180720_1221.py
swh/deposit/migrations/0015_depositrequest_typemigration.py
swh/deposit/migrations/0016_auto_20190507_1408.py
swh/deposit/migrations/0017_auto_20190925_0906.py
swh/deposit/migrations/0018_migrate_swhids.py
swh/deposit/migrations/0019_auto_20200519_1035.py
swh/deposit/migrations/0020_auto_20200929_0855.py
swh/deposit/migrations/__init__.py
swh/deposit/settings/__init__.py
swh/deposit/settings/common.py
swh/deposit/settings/development.py
swh/deposit/settings/production.py
swh/deposit/settings/testing.py
swh/deposit/static/robots.txt
swh/deposit/static/css/bootstrap-responsive.min.css
swh/deposit/static/css/style.css
swh/deposit/static/img/arrow-up-small.png
swh/deposit/static/img/swh-logo-deposit.png
swh/deposit/static/img/swh-logo-deposit.svg
swh/deposit/static/img/icons/swh-logo-32x32.png
swh/deposit/static/img/icons/swh-logo-deposit-180x180.png
swh/deposit/static/img/icons/swh-logo-deposit-192x192.png
swh/deposit/static/img/icons/swh-logo-deposit-270x270.png
swh/deposit/templates/__init__.py
swh/deposit/templates/api.html
swh/deposit/templates/homepage.html
swh/deposit/templates/layout.html
swh/deposit/templates/deposit/__init__.py
swh/deposit/templates/deposit/content.xml
swh/deposit/templates/deposit/deposit_receipt.xml
swh/deposit/templates/deposit/error.xml
swh/deposit/templates/deposit/service_document.xml
swh/deposit/templates/deposit/status.xml
swh/deposit/templates/rest_framework/api.html
swh/deposit/tests/__init__.py
swh/deposit/tests/common.py
swh/deposit/tests/conftest.py
swh/deposit/tests/test_common.py
swh/deposit/tests/test_gunicorn_config.py
swh/deposit/tests/test_init.py
swh/deposit/tests/test_utils.py
swh/deposit/tests/api/__init__.py
swh/deposit/tests/api/conftest.py
swh/deposit/tests/api/test_checks.py
swh/deposit/tests/api/test_converters.py
swh/deposit/tests/api/test_deposit.py
swh/deposit/tests/api/test_deposit_atom.py
swh/deposit/tests/api/test_deposit_binary.py
+swh/deposit/tests/api/test_deposit_content.py
swh/deposit/tests/api/test_deposit_delete.py
swh/deposit/tests/api/test_deposit_list.py
swh/deposit/tests/api/test_deposit_multipart.py
swh/deposit/tests/api/test_deposit_private_check.py
swh/deposit/tests/api/test_deposit_private_read_archive.py
swh/deposit/tests/api/test_deposit_private_read_metadata.py
swh/deposit/tests/api/test_deposit_private_update_status.py
swh/deposit/tests/api/test_deposit_schedule.py
swh/deposit/tests/api/test_deposit_status.py
swh/deposit/tests/api/test_deposit_update.py
swh/deposit/tests/api/test_exception.py
swh/deposit/tests/api/test_parser.py
swh/deposit/tests/api/test_service_document.py
swh/deposit/tests/cli/__init__.py
swh/deposit/tests/cli/test_client.py
swh/deposit/tests/data/atom/codemeta-sample.xml
swh/deposit/tests/data/atom/entry-data-badly-formatted.xml
swh/deposit/tests/data/atom/entry-data-deposit-binary.xml
swh/deposit/tests/data/atom/entry-data-empty-body-no-namespace.xml
swh/deposit/tests/data/atom/entry-data-empty-body.xml
swh/deposit/tests/data/atom/entry-data-fail-metadata-functional-checks.xml
swh/deposit/tests/data/atom/entry-data-ko.xml
swh/deposit/tests/data/atom/entry-data-minimal.xml
swh/deposit/tests/data/atom/entry-data-parsing-error-prone.xml
swh/deposit/tests/data/atom/entry-data0.xml
swh/deposit/tests/data/atom/entry-data1.xml
swh/deposit/tests/data/atom/entry-data2.xml
swh/deposit/tests/data/atom/entry-data3.xml
swh/deposit/tests/data/atom/entry-update-in-place.xml
swh/deposit/tests/data/atom/error-with-decimal.xml
swh/deposit/tests/data/atom/metadata.xml
swh/deposit/tests/data/atom/tei-sample.xml
+swh/deposit/tests/data/https_deposit.swh.test/1_servicedocument
+swh/deposit/tests/data/https_deposit.swh.test/1_test
+swh/deposit/tests/data/https_deposit.test.metadata/1_servicedocument
+swh/deposit/tests/data/https_deposit.test.metadata/1_test
+swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_media
+swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_metadata
+swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_status
+swh/deposit/tests/data/https_deposit.test.status/1_servicedocument
+swh/deposit/tests/data/https_deposit.test.status/1_test_1033_status
+swh/deposit/tests/data/https_deposit.test.updateswhid/1_servicedocument
+swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_123_metadata
+swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_123_status
+swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_321_status
swh/deposit/tests/loader/__init__.py
swh/deposit/tests/loader/common.py
swh/deposit/tests/loader/conftest.py
swh/deposit/tests/loader/test_checker.py
swh/deposit/tests/loader/test_client.py
swh/deposit/tests/loader/test_tasks.py
swh/deposit/tests/loader/data/http_example.org/hello.json
swh/deposit/tests/loader/data/http_example.org/hello_you
swh/deposit/tests/loader/data/https_deposit.softwareheritage.org/1_private_test_1_check
swh/deposit/tests/loader/data/https_deposit.softwareheritage.org/1_private_test_2_check
swh/deposit/tests/loader/data/https_deposit.softwareheritage.org/1_private_test_999_meta
swh/deposit/tests/loader/data/https_deposit.softwareheritage.org/1_private_test_999_raw
swh/deposit/tests/loader/data/https_deposit.softwareheritage.org/1_private_test_999_update
swh/deposit/tests/loader/data/https_nowhere.org/1_private_test_1_check
swh/deposit/tests/loader/data/https_nowhere.org/1_private_test_1_metadata
swh/deposit/tests/loader/data/https_nowhere.org/1_private_test_1_raw
\ No newline at end of file
diff --git a/swh.deposit.egg-info/requires.txt b/swh.deposit.egg-info/requires.txt
index 5c5013a6..1efd36b0 100644
--- a/swh.deposit.egg-info/requires.txt
+++ b/swh.deposit.egg-info/requires.txt
@@ -1,31 +1,31 @@
click
xmltodict
iso8601
requests
-swh.core[http]>=0.3
+swh.core[http]>=0.4
[server]
Django<3
djangorestframework
setuptools
-swh.core[http]
+swh.core[http]>=0.4
swh.loader.core>=0.0.71
swh.scheduler>=0.0.39
swh.model>=0.3.8
[testing]
pytest
pytest-django
pytest-mock
swh.scheduler[testing]
swh.loader.core[testing]
pytest-postgresql>=2.1.0
requests_mock
django-stubs
Django<3
djangorestframework
setuptools
-swh.core[http]
+swh.core[http]>=0.4
swh.loader.core>=0.0.71
swh.scheduler>=0.0.39
swh.model>=0.3.8
diff --git a/swh/deposit/cli/client.py b/swh/deposit/cli/client.py
index 1e68d8c4..1f906be4 100644
--- a/swh/deposit/cli/client.py
+++ b/swh/deposit/cli/client.py
@@ -1,509 +1,530 @@
# Copyright (C) 2017-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
+from __future__ import annotations
+
import logging
# WARNING: do not import unnecessary things here to keep cli startup time under
# control
import os
import sys
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
import click
from swh.deposit.cli import deposit
logger = logging.getLogger(__name__)
+if TYPE_CHECKING:
+ from swh.deposit.client import PublicApiDepositClient
+
+
class InputError(ValueError):
"""Input script error
"""
pass
-def generate_slug():
+def generate_slug() -> str:
"""Generate a slug (sample purposes).
"""
import uuid
return str(uuid.uuid4())
-def _url(url):
+def _url(url: str) -> str:
"""Force the /1 api version at the end of the url (avoiding confusing
issues without it).
Args:
url (str): api url used by cli users
Returns:
Top level api url to actually request
"""
if not url.endswith("/1"):
url = "%s/1" % url
return url
-def generate_metadata_file(name, external_id, authors, temp_dir):
+def generate_metadata_file(
+ name: str, external_id: str, authors: List[str], temp_dir: str
+) -> str:
"""Generate a temporary metadata file with the minimum required metadata
This generates a xml file in a temporary location and returns the
path to that file.
This is up to the client of that function to clean up the
temporary file.
Args:
- name (str): Software's name
- external_id (str): External identifier (slug) or generated one
- authors (List[str]): List of author names
+ name: Software's name
+ external_id: External identifier (slug) or generated one
+ authors: List of author names
Returns:
Filepath to the metadata generated file
"""
import xmltodict
path = os.path.join(temp_dir, "metadata.xml")
# generate a metadata file with the minimum required metadata
codemetadata = {
"entry": {
"@xmlns": "http://www.w3.org/2005/Atom",
"@xmlns:codemeta": "https://doi.org/10.5063/SCHEMA/CODEMETA-2.0",
"codemeta:name": name,
"codemeta:identifier": external_id,
"codemeta:author": [
{"codemeta:name": author_name} for author_name in authors
],
},
}
logging.debug("Temporary file: %s", path)
logging.debug("Metadata dict to generate as xml: %s", codemetadata)
s = xmltodict.unparse(codemetadata, pretty=True)
logging.debug("Metadata dict as xml generated: %s", s)
with open(path, "w") as fp:
fp.write(s)
return path
-def _client(url, username, password):
+def _client(url: str, username: str, password: str) -> PublicApiDepositClient:
"""Instantiate a client to access the deposit api server
Args:
url (str): Deposit api server
username (str): User
password (str): User's password
"""
from swh.deposit.client import PublicApiDepositClient
- client = PublicApiDepositClient(
+ return PublicApiDepositClient(
{"url": url, "auth": {"username": username, "password": password},}
)
- return client
-def _collection(client):
+def _collection(client: PublicApiDepositClient) -> str:
"""Retrieve the client's collection
"""
# retrieve user's collection
sd_content = client.service_document()
if "error" in sd_content:
raise InputError("Service document retrieval: %s" % (sd_content["error"],))
collection = sd_content["service"]["workspace"]["collection"]["sword:name"]
return collection
def client_command_parse_input(
- username,
- password,
- archive,
- metadata,
- archive_deposit,
- metadata_deposit,
- collection,
- slug,
- partial,
- deposit_id,
- replace,
- url,
- name,
- authors,
- temp_dir,
-):
+ username: str,
+ password: str,
+ archive: Optional[str],
+ metadata: Optional[str],
+ archive_deposit: bool,
+ metadata_deposit: bool,
+ collection: Optional[str],
+ slug: Optional[str],
+ partial: bool,
+ deposit_id: Optional[int],
+ swhid: Optional[str],
+ replace: bool,
+ url: str,
+ name: Optional[str],
+ authors: List[str],
+ temp_dir: str,
+) -> Dict[str, Any]:
"""Parse the client subcommand options and make sure the combination
is acceptable*. If not, an InputError exception is raised
explaining the issue.
By acceptable, we mean:
- A multipart deposit (create or update) requires:
- an existing software archive
- an existing metadata file or author(s) and name provided in
params
- A binary deposit (create/update) requires an existing software
archive
- A metadata deposit (create/update) requires an existing metadata
file or author(s) and name provided in params
- A deposit update requires a deposit_id
This will not prevent all failure cases though. The remaining
errors are already dealt with by the underlying api client.
Raises:
InputError explaining the user input related issue
MaintenanceError explaining the api status
Returns:
dict with the following keys:
'archive': the software archive to deposit
'username': username
'password': associated password
'metadata': the metadata file to deposit
'collection': the username's associated client
'slug': the slug or external id identifying the deposit to make
'partial': if the deposit is partial or not
'client': instantiated class
'url': deposit's server main entry point
'deposit_type': deposit's type (binary, multipart, metadata)
'deposit_id': optional deposit identifier
+ 'swhid': optional deposit swhid
"""
if archive_deposit and metadata_deposit:
# too many flags use, remove redundant ones (-> multipart deposit)
archive_deposit = False
metadata_deposit = False
if not slug: # generate one as this is mandatory
slug = generate_slug()
if not metadata:
+ if metadata_deposit:
+ raise InputError(
+ "Metadata deposit must be provided for metadata "
+ "deposit, either a filepath with --metadata or --name and --author"
+ )
if name and authors:
metadata = generate_metadata_file(name, slug, authors, temp_dir)
elif not archive_deposit and not partial and not deposit_id:
# If we meet all the following conditions:
- # * there is not an archive-only deposit
+ # * this is not an archive-only deposit request
# * it is not part of a multipart deposit (either create/update
# or finish)
# * it misses either name or authors
raise InputError(
- "Either a metadata file (--metadata) or both --author and "
- "--name must be provided, unless this is an archive-only "
- "deposit."
+ "For metadata deposit request, either a metadata file with "
+ "--metadata or both --author and --name must be provided. "
+ "If this is an archive deposit request, none is required."
)
elif name or authors:
# If we are generating metadata, then all mandatory metadata
# must be present
raise InputError(
- "Either a metadata file (--metadata) or both --author and "
- "--name must be provided."
+ "For metadata deposit request, either a metadata file with "
+ "--metadata or both --author and --name must be provided."
)
else:
# TODO: this is a multipart deposit, we might want to check that
# metadata are deposited at some point
pass
elif name or authors:
raise InputError(
- "Using a metadata file (--metadata) is incompatible with "
- "--author and --name, which are used to generate one."
+ "Using --metadata flag is incompatible with both "
+ "--author and --name (Those are used to generate one metadata file)."
)
if metadata_deposit:
archive = None
if archive_deposit:
metadata = None
- if metadata_deposit and not metadata:
- raise InputError(
- "Metadata deposit must be provided for metadata "
- "deposit (either a filepath or --name and --author)"
- )
-
if not archive and not metadata and partial:
raise InputError(
"Please provide an actionable command. See --help for more information"
)
if replace and not deposit_id:
raise InputError("To update an existing deposit, you must provide its id")
client = _client(url, username, password)
if not collection:
collection = _collection(client)
return {
"archive": archive,
"username": username,
"password": password,
"metadata": metadata,
"collection": collection,
"slug": slug,
"in_progress": partial,
"client": client,
"url": url,
"deposit_id": deposit_id,
+ "swhid": swhid,
"replace": replace,
}
-def _subdict(d, keys):
+def _subdict(d: Dict[str, Any], keys: Tuple[str, ...]) -> Dict[str, Any]:
"return a dict from d with only given keys"
return {k: v for k, v in d.items() if k in keys}
-def deposit_create(config, logger):
+def deposit_create(config: Dict[str, Any]) -> Dict[str, Any]:
"""Delegate the actual deposit to the deposit client.
"""
logger.debug("Create deposit")
client = config["client"]
keys = ("collection", "archive", "metadata", "slug", "in_progress")
return client.deposit_create(**_subdict(config, keys))
-def deposit_update(config, logger):
+def deposit_update(config: Dict[str, Any]) -> Dict[str, Any]:
"""Delegate the actual deposit to the deposit client.
"""
logger.debug("Update deposit")
client = config["client"]
keys = (
"collection",
"deposit_id",
"archive",
"metadata",
"slug",
"in_progress",
"replace",
+ "swhid",
)
return client.deposit_update(**_subdict(config, keys))
@deposit.command()
@click.option("--username", required=True, help="(Mandatory) User's name")
@click.option(
"--password", required=True, help="(Mandatory) User's associated password"
)
@click.option(
"--archive",
type=click.Path(exists=True),
help="(Optional) Software archive to deposit",
)
@click.option(
"--metadata",
type=click.Path(exists=True),
help=(
"(Optional) Path to xml metadata file. If not provided, "
"this will use a file named <archive>.metadata.xml"
),
) # noqa
@click.option(
"--archive-deposit/--no-archive-deposit",
default=False,
help="(Optional) Software archive only deposit",
)
@click.option(
"--metadata-deposit/--no-metadata-deposit",
default=False,
help="(Optional) Metadata only deposit",
)
@click.option(
"--collection",
help="(Optional) User's collection. If not provided, this will be fetched.",
) # noqa
@click.option(
"--slug",
help=(
"(Optional) External system information identifier. "
"If not provided, it will be generated"
),
) # noqa
@click.option(
"--partial/--no-partial",
default=False,
help=(
"(Optional) The deposit will be partial, other deposits "
"will have to take place to finalize it."
),
) # noqa
@click.option(
"--deposit-id",
default=None,
help="(Optional) Update an existing partial deposit with its identifier",
) # noqa
+@click.option(
+ "--swhid",
+ default=None,
+ help="(Optional) Update existing completed deposit (status done) with new metadata",
+)
@click.option(
"--replace/--no-replace",
default=False,
help="(Optional) Update by replacing existing metadata to a deposit",
) # noqa
@click.option(
"--url",
default="https://deposit.softwareheritage.org",
help=(
"(Optional) Deposit server api endpoint. By default, "
"https://deposit.softwareheritage.org/1"
),
) # noqa
@click.option("--verbose/--no-verbose", default=False, help="Verbose mode")
@click.option("--name", help="Software name")
@click.option(
"--author",
multiple=True,
help="Software author(s), this can be repeated as many times"
" as there are authors",
)
@click.option(
"-f",
"--format",
"output_format",
default="logging",
type=click.Choice(["logging", "yaml", "json"]),
help="Output format results.",
)
@click.pass_context
def upload(
ctx,
- username,
- password,
- archive=None,
- metadata=None,
- archive_deposit=False,
- metadata_deposit=False,
- collection=None,
- slug=None,
- partial=False,
- deposit_id=None,
- replace=False,
- url="https://deposit.softwareheritage.org",
- verbose=False,
- name=None,
- author=None,
- output_format=None,
+ username: str,
+ password: str,
+ archive: Optional[str] = None,
+ metadata: Optional[str] = None,
+ archive_deposit: bool = False,
+ metadata_deposit: bool = False,
+ collection: Optional[str] = None,
+ slug: Optional[str] = None,
+ partial: bool = False,
+ deposit_id: Optional[int] = None,
+ swhid: Optional[str] = None,
+ replace: bool = False,
+ url: str = "https://deposit.softwareheritage.org",
+ verbose: bool = False,
+ name: Optional[str] = None,
+ author: List[str] = [],
+ output_format: Optional[str] = None,
):
"""Software Heritage Public Deposit Client
Create/Update deposit through the command line.
More documentation can be found at
https://docs.softwareheritage.org/devel/swh-deposit/getting-started.html.
"""
import tempfile
from swh.deposit.client import MaintenanceError
url = _url(url)
config = {}
with tempfile.TemporaryDirectory() as temp_dir:
try:
logger.debug("Parsing cli options")
config = client_command_parse_input(
username,
password,
archive,
metadata,
archive_deposit,
metadata_deposit,
collection,
slug,
partial,
deposit_id,
+ swhid,
replace,
url,
name,
author,
temp_dir,
)
except InputError as e:
logger.error("Problem during parsing options: %s", e)
sys.exit(1)
except MaintenanceError as e:
logger.error(e)
sys.exit(1)
if verbose:
- logger.info("Parsed configuration: %s" % (config,))
+ logger.info("Parsed configuration: %s", config)
deposit_id = config["deposit_id"]
if deposit_id:
- r = deposit_update(config, logger)
+ data = deposit_update(config)
else:
- r = deposit_create(config, logger)
- print_result(r, output_format)
+ data = deposit_create(config)
+ print_result(data, output_format)
@deposit.command()
@click.option(
"--url",
default="https://deposit.softwareheritage.org",
help="(Optional) Deposit server api endpoint. By default, "
"https://deposit.softwareheritage.org/1",
)
@click.option("--username", required=True, help="(Mandatory) User's name")
@click.option(
"--password", required=True, help="(Mandatory) User's associated password"
)
@click.option("--deposit-id", default=None, required=True, help="Deposit identifier.")
@click.option(
"-f",
"--format",
"output_format",
default="logging",
type=click.Choice(["logging", "yaml", "json"]),
help="Output format results.",
)
@click.pass_context
def status(ctx, url, username, password, deposit_id, output_format):
"""Deposit's status
"""
from swh.deposit.client import MaintenanceError
url = _url(url)
logger.debug("Status deposit")
try:
client = _client(url, username, password)
collection = _collection(client)
except InputError as e:
logger.error("Problem during parsing options: %s", e)
sys.exit(1)
except MaintenanceError as e:
logger.error(e)
sys.exit(1)
print_result(
client.deposit_status(collection=collection, deposit_id=deposit_id),
output_format,
)
-def print_result(data, output_format):
+def print_result(data: Dict[str, Any], output_format: Optional[str]) -> None:
+ """Display the result data into a dedicated output format.
+
+ """
import json
import yaml
if output_format == "json":
click.echo(json.dumps(data))
elif output_format == "yaml":
click.echo(yaml.dump(data))
else:
logger.info(data)
diff --git a/swh/deposit/client.py b/swh/deposit/client.py
index a27c166b..d6d36c06 100644
--- a/swh/deposit/client.py
+++ b/swh/deposit/client.py
@@ -1,651 +1,712 @@
# Copyright (C) 2017-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
"""Module in charge of defining an swh-deposit client
"""
from abc import ABCMeta, abstractmethod
import hashlib
import logging
import os
-from typing import Any, Dict
+from typing import Any, Dict, Optional
from urllib.parse import urljoin
import requests
import xmltodict
-from swh.core.config import config_basepath, read_raw_config
+from swh.core.config import load_from_envvar
logger = logging.getLogger(__name__)
+def compute_unified_information(
+ collection: str,
+ in_progress: bool,
+ slug: str,
+ *,
+ filepath: Optional[str] = None,
+ swhid: Optional[str] = None,
+ **kwargs,
+) -> Dict[str, Any]:
+ """Given a filepath, compute necessary information on that file.
+
+ Args:
+ collection: Deposit collection
+ in_progress: do we finalize the deposit?
+ slug: external id to use
+ filepath: Path to the file to compute the necessary information out of
+ swhid: Deposit swhid if any
+
+ Returns:
+ dict with keys:
+
+ 'slug': external id to use
+ 'in_progress': do we finalize the deposit?
+ 'content-type': content type associated
+ 'md5sum': md5 sum
+ 'filename': filename
+ 'filepath': filepath
+ 'swhid': deposit swhid
+
+ """
+ result: Dict[str, Any] = {
+ "slug": slug,
+ "in_progress": in_progress,
+ "swhid": swhid,
+ }
+ content_type: Optional[str] = None
+ md5sum: Optional[str] = None
+
+ if filepath:
+ filename = os.path.basename(filepath)
+ md5sum = hashlib.md5(open(filepath, "rb").read()).hexdigest()
+ extension = filename.split(".")[-1]
+ if "zip" in extension:
+ content_type = "application/zip"
+ else:
+ content_type = "application/x-tar"
+ result.update(
+ {
+ "content-type": content_type,
+ "md5sum": md5sum,
+ "filename": filename,
+ "filepath": filepath,
+ }
+ )
+
+ return result
+
+
class MaintenanceError(ValueError):
"""Informational maintenance error exception
"""
pass
def _parse(stream, encoding="utf-8"):
"""Given a xml stream, parse the result.
Args:
stream (bytes/text): The stream to parse
encoding (str): The encoding to use if to decode the bytes
stream
Returns:
A dict of values corresponding to the parsed xml
"""
if isinstance(stream, bytes):
stream = stream.decode(encoding)
data = xmltodict.parse(stream, encoding=encoding, process_namespaces=False)
if "entry" in data:
data = data["entry"]
if "sword:error" in data:
data = data["sword:error"]
return dict(data)
def _parse_with_filter(stream, encoding="utf-8", keys=[]):
"""Given a xml stream, parse the result and filter with keys.
Args:
stream (bytes/text): The stream to parse
encoding (str): The encoding to use if to decode the bytes
stream
keys ([str]): Keys to filter the parsed result
Returns:
A dict of values corresponding to the parsed xml filtered by
the keys provided.
"""
data = _parse(stream, encoding=encoding)
m = {}
for key in keys:
m[key] = data.get(key)
return m
class BaseApiDepositClient:
"""Deposit client base class
"""
def __init__(self, config=None, _client=requests):
- if config is None:
- config_file = os.environ["SWH_CONFIG_FILENAME"]
- self.config: Dict[str, Any] = read_raw_config(config_basepath(config_file))
- else:
- self.config = config
-
+ self.config: Dict[str, Any] = config or load_from_envvar()
self._client = _client
self.base_url = self.config["url"].strip("/") + "/"
auth = self.config["auth"]
if auth == {}:
self.auth = None
else:
self.auth = (auth["username"], auth["password"])
def do(self, method, url, *args, **kwargs):
"""Internal method to deal with requests, possibly with basic http
authentication.
Args:
method (str): supported http methods as in self._methods' keys
Returns:
The request's execution
"""
if hasattr(self._client, method):
method_fn = getattr(self._client, method)
else:
raise ValueError("Development error, unsupported method %s" % (method))
if self.auth:
kwargs["auth"] = self.auth
full_url = urljoin(self.base_url, url.lstrip("/"))
return method_fn(full_url, *args, **kwargs)
class PrivateApiDepositClient(BaseApiDepositClient):
"""Private API deposit client to:
- read a given deposit's archive(s)
- read a given deposit's metadata
- update a given deposit's status
"""
def archive_get(self, archive_update_url, archive):
"""Retrieve the archive from the deposit to a local directory.
Args:
archive_update_url (str): The full deposit archive(s)'s raw content
to retrieve locally
archive (str): the local archive's path where to store
the raw content
Returns:
The archive path to the local archive to load.
Or None if any problem arose.
"""
r = self.do("get", archive_update_url, stream=True)
if r.ok:
with open(archive, "wb") as f:
for chunk in r.iter_content():
f.write(chunk)
return archive
msg = "Problem when retrieving deposit archive at %s" % (archive_update_url,)
logger.error(msg)
raise ValueError(msg)
def metadata_get(self, metadata_url):
"""Retrieve the metadata information on a given deposit.
Args:
metadata_url (str): The full deposit metadata url to retrieve
locally
Returns:
The dictionary of metadata for that deposit or None if any
problem arose.
"""
r = self.do("get", metadata_url)
if r.ok:
return r.json()
msg = "Problem when retrieving metadata at %s" % metadata_url
logger.error(msg)
raise ValueError(msg)
def status_update(
self,
update_status_url,
status,
revision_id=None,
directory_id=None,
origin_url=None,
):
"""Update the deposit's status.
Args:
update_status_url (str): the full deposit's archive
status (str): The status to update the deposit with
revision_id (str/None): the revision's identifier to update to
directory_id (str/None): the directory's identifier to update to
origin_url (str/None): deposit's associated origin url
"""
payload = {"status": status}
if revision_id:
payload["revision_id"] = revision_id
if directory_id:
payload["directory_id"] = directory_id
if origin_url:
payload["origin_url"] = origin_url
self.do("put", update_status_url, json=payload)
def check(self, check_url):
"""Check the deposit's associated data (metadata, archive(s))
Args:
check_url (str): the full deposit's check url
"""
r = self.do("get", check_url)
if r.ok:
data = r.json()
return data["status"]
msg = "Problem when checking deposit %s" % check_url
logger.error(msg)
raise ValueError(msg)
class BaseDepositClient(BaseApiDepositClient, metaclass=ABCMeta):
"""Base Deposit client to access the public api.
"""
def __init__(self, config, error_msg=None, empty_result={}):
super().__init__(config)
self.error_msg = error_msg
self.empty_result = empty_result
@abstractmethod
def compute_url(self, *args, **kwargs):
"""Compute api url endpoint to query."""
pass
@abstractmethod
def compute_method(self, *args, **kwargs):
"""Http method to use on the url"""
pass
@abstractmethod
def parse_result_ok(self, xml_content):
"""Given an xml result from the api endpoint, parse it and returns a
dict.
"""
pass
- def compute_information(self, *args, **kwargs):
+ def compute_information(self, *args, **kwargs) -> Dict[str, Any]:
"""Compute some more information given the inputs (e.g http headers,
...)
"""
return {}
def parse_result_error(self, xml_content):
"""Given an error response in xml, parse it into a dict.
Returns:
dict with following keys:
'error': The error message
'detail': Some more detail about the error if any
"""
return _parse_with_filter(
xml_content, keys=["summary", "detail", "sword:verboseDescription"]
)
def do_execute(self, method, url, info):
"""Execute the http query to url using method and info information.
By default, execute a simple query to url with the http
method. Override this in daughter class to improve the
default behavior if needed.
"""
return self.do(method, url)
def execute(self, *args, **kwargs) -> Dict[str, Any]:
"""Main endpoint to prepare and execute the http query to the api.
Raises:
MaintenanceError if some api maintenance is happening.
Returns:
Dict of computed api data
"""
url = self.compute_url(*args, **kwargs)
method = self.compute_method(*args, **kwargs)
info = self.compute_information(*args, **kwargs)
try:
r = self.do_execute(method, url, info)
except Exception as e:
msg = self.error_msg % (url, e)
r = self.empty_result
r.update(
{"error": msg,}
)
return r
else:
if r.ok:
if int(r.status_code) == 204: # 204 returns no body
return {"status": r.status_code}
else:
return self.parse_result_ok(r.text)
else:
error = self.parse_result_error(r.text)
empty = self.empty_result
error.update(empty)
if r.status_code == 503:
summary = error.get("summary")
detail = error.get("sword:verboseDescription")
# Maintenance error
if summary and detail:
raise MaintenanceError(f"{summary}: {detail}")
error.update(
{"status": r.status_code,}
)
return error
class ServiceDocumentDepositClient(BaseDepositClient):
"""Service Document information retrieval.
"""
def __init__(self, config):
super().__init__(
config,
error_msg="Service document failure at %s: %s",
empty_result={"collection": None},
)
def compute_url(self, *args, **kwargs):
return "/servicedocument/"
def compute_method(self, *args, **kwargs):
return "get"
def parse_result_ok(self, xml_content):
"""Parse service document's success response.
"""
return _parse(xml_content)
class StatusDepositClient(BaseDepositClient):
"""Status information on a deposit.
"""
def __init__(self, config):
super().__init__(
config,
error_msg="Status check failure at %s: %s",
empty_result={
"deposit_status": None,
"deposit_status_detail": None,
"deposit_swh_id": None,
},
)
def compute_url(self, collection, deposit_id):
return "/%s/%s/status/" % (collection, deposit_id)
def compute_method(self, *args, **kwargs):
return "get"
def parse_result_ok(self, xml_content):
"""Given an xml content as string, returns a deposit dict.
"""
return _parse_with_filter(
xml_content,
keys=[
"deposit_id",
"deposit_status",
"deposit_status_detail",
"deposit_swh_id",
"deposit_swh_id_context",
"deposit_external_id",
],
)
class BaseCreateDepositClient(BaseDepositClient):
"""Deposit client base class to post new deposit.
"""
def __init__(self, config):
super().__init__(
config,
error_msg="Post Deposit failure at %s: %s",
empty_result={"deposit_id": None, "deposit_status": None,},
)
def compute_url(self, collection, *args, **kwargs):
return "/%s/" % collection
def compute_method(self, *args, **kwargs):
return "post"
def parse_result_ok(self, xml_content):
"""Given an xml content as string, returns a deposit dict.
"""
return _parse_with_filter(
xml_content,
keys=[
"deposit_id",
"deposit_status",
"deposit_status_detail",
"deposit_date",
],
)
- def _compute_information(
- self, collection, filepath, in_progress, slug, is_archive=True
- ):
- """Given a filepath, compute necessary information on that file.
-
- Args:
- filepath (str): Path to a file
- is_archive (bool): is it an archive or not?
-
- Returns:
- dict with keys:
- 'content-type': content type associated
- 'md5sum': md5 sum
- 'filename': filename
- """
- filename = os.path.basename(filepath)
-
- if is_archive:
- md5sum = hashlib.md5(open(filepath, "rb").read()).hexdigest()
- extension = filename.split(".")[-1]
- if "zip" in extension:
- content_type = "application/zip"
- else:
- content_type = "application/x-tar"
- else:
- content_type = None
- md5sum = None
-
- return {
- "slug": slug,
- "in_progress": in_progress,
- "content-type": content_type,
- "md5sum": md5sum,
- "filename": filename,
- "filepath": filepath,
- }
-
- def compute_information(
- self, collection, filepath, in_progress, slug, is_archive=True, **kwargs
- ):
- info = self._compute_information(
- collection, filepath, in_progress, slug, is_archive=is_archive
- )
- info["headers"] = self.compute_headers(info)
+ def compute_headers(self, info: Dict[str, Any]) -> Dict[str, Any]:
return info
def do_execute(self, method, url, info):
with open(info["filepath"], "rb") as f:
return self.do(method, url, data=f, headers=info["headers"])
class CreateArchiveDepositClient(BaseCreateDepositClient):
"""Post an archive (binary) deposit client."""
def compute_headers(self, info):
return {
"SLUG": info["slug"],
"CONTENT_MD5": info["md5sum"],
"IN-PROGRESS": str(info["in_progress"]),
"CONTENT-TYPE": info["content-type"],
"CONTENT-DISPOSITION": "attachment; filename=%s" % (info["filename"],),
}
+ def compute_information(self, *args, **kwargs) -> Dict[str, Any]:
+ info = compute_unified_information(
+ *args, filepath=kwargs["archive_path"], **kwargs
+ )
+ info["headers"] = self.compute_headers(info)
+ return info
+
class UpdateArchiveDepositClient(CreateArchiveDepositClient):
"""Update (add/replace) an archive (binary) deposit client."""
def compute_url(self, collection, *args, deposit_id=None, **kwargs):
return "/%s/%s/media/" % (collection, deposit_id)
def compute_method(self, *args, replace=False, **kwargs):
return "put" if replace else "post"
class CreateMetadataDepositClient(BaseCreateDepositClient):
"""Post a metadata deposit client."""
def compute_headers(self, info):
return {
"SLUG": info["slug"],
"IN-PROGRESS": str(info["in_progress"]),
"CONTENT-TYPE": "application/atom+xml;type=entry",
}
+ def compute_information(self, *args, **kwargs) -> Dict[str, Any]:
+ info = compute_unified_information(
+ *args, filepath=kwargs["metadata_path"], **kwargs
+ )
+ info["headers"] = self.compute_headers(info)
+ return info
+
-class UpdateMetadataDepositClient(CreateMetadataDepositClient):
- """Update (add/replace) a metadata deposit client."""
+class UpdateMetadataOnPartialDepositClient(CreateMetadataDepositClient):
+ """Update (add/replace) metadata on partial deposit scenario."""
def compute_url(self, collection, *args, deposit_id=None, **kwargs):
- return "/%s/%s/metadata/" % (collection, deposit_id)
+ return f"/{collection}/{deposit_id}/metadata/"
- def compute_method(self, *args, replace=False, **kwargs):
+ def compute_method(self, *args, replace: bool = False, **kwargs) -> str:
return "put" if replace else "post"
+class UpdateMetadataOnDoneDepositClient(UpdateMetadataOnPartialDepositClient):
+ """Update metadata on "done" deposit. This requires the deposit swhid."""
+
+ def compute_headers(self, info: Dict[str, Any]) -> Dict[str, Any]:
+ return {
+ "CONTENT-TYPE": "application/atom+xml;type=entry",
+ "X_CHECK_SWHID": info["swhid"],
+ }
+
+ def compute_method(self, *args, **kwargs) -> str:
+ return "put"
+
+
class CreateMultipartDepositClient(BaseCreateDepositClient):
"""Create a multipart deposit client."""
def _multipart_info(self, info, info_meta):
files = [
(
"file",
(info["filename"], open(info["filepath"], "rb"), info["content-type"]),
),
(
"atom",
(
info_meta["filename"],
open(info_meta["filepath"], "rb"),
"application/atom+xml",
),
),
]
headers = {
"SLUG": info["slug"],
"CONTENT_MD5": info["md5sum"],
"IN-PROGRESS": str(info["in_progress"]),
}
return files, headers
- def compute_information(
- self, collection, archive, metadata, in_progress, slug, **kwargs
- ):
- info = self._compute_information(collection, archive, in_progress, slug)
- info_meta = self._compute_information(
- collection, metadata, in_progress, slug, is_archive=False
+ def compute_information(self, *args, **kwargs) -> Dict[str, Any]:
+ info = compute_unified_information(*args, filepath=kwargs["archive_path"],)
+ info_meta = compute_unified_information(
+ *args, filepath=kwargs["metadata_path"],
)
files, headers = self._multipart_info(info, info_meta)
return {"files": files, "headers": headers}
def do_execute(self, method, url, info):
return self.do(method, url, files=info["files"], headers=info["headers"])
class UpdateMultipartDepositClient(CreateMultipartDepositClient):
"""Update a multipart deposit client."""
def compute_url(self, collection, *args, deposit_id=None, **kwargs):
return "/%s/%s/metadata/" % (collection, deposit_id)
def compute_method(self, *args, replace=False, **kwargs):
return "put" if replace else "post"
class PublicApiDepositClient(BaseApiDepositClient):
"""Public api deposit client."""
def service_document(self):
"""Retrieve service document endpoint's information."""
return ServiceDocumentDepositClient(self.config).execute()
- def deposit_status(self, collection, deposit_id):
+ def deposit_status(self, collection: str, deposit_id: int):
"""Retrieve status information on a deposit."""
return StatusDepositClient(self.config).execute(collection, deposit_id)
def deposit_create(
- self, collection, slug, archive=None, metadata=None, in_progress=False
+ self,
+ collection: str,
+ slug: str,
+ archive: Optional[str] = None,
+ metadata: Optional[str] = None,
+ in_progress: bool = False,
):
"""Create a new deposit (archive, metadata, both as multipart)."""
if archive and not metadata:
return CreateArchiveDepositClient(self.config).execute(
- collection, archive, in_progress, slug
+ collection, in_progress, slug, archive_path=archive
)
elif not archive and metadata:
return CreateMetadataDepositClient(self.config).execute(
- collection, metadata, in_progress, slug, is_archive=False
+ collection, in_progress, slug, metadata_path=metadata
)
else:
return CreateMultipartDepositClient(self.config).execute(
- collection, archive, metadata, in_progress, slug
+ collection,
+ in_progress,
+ slug,
+ archive_path=archive,
+ metadata_path=metadata,
)
def deposit_update(
self,
- collection,
- deposit_id,
- slug,
- archive=None,
- metadata=None,
- in_progress=False,
- replace=False,
+ collection: str,
+ deposit_id: int,
+ slug: str,
+ archive: Optional[str] = None,
+ metadata: Optional[str] = None,
+ in_progress: bool = False,
+ replace: bool = False,
+ swhid: Optional[str] = None,
):
"""Update (add/replace) existing deposit (archive, metadata, both)."""
r = self.deposit_status(collection, deposit_id)
if "error" in r:
return r
status = r["deposit_status"]
- if status != "partial":
+ if swhid is None and status != "partial":
return {
"error": "You can only act on deposit with status 'partial'",
- "detail": "The deposit %s has status '%s'" % (deposit_id, status),
+ "detail": f"The deposit {deposit_id} has status '{status}'",
+ "deposit_status": status,
+ "deposit_id": deposit_id,
+ }
+ if swhid is not None and status != "done":
+ return {
+ "error": "You can only update metadata on deposit with status 'done'",
+ "detail": f"The deposit {deposit_id} has status '{status}'",
"deposit_status": status,
"deposit_id": deposit_id,
}
if archive and not metadata:
r = UpdateArchiveDepositClient(self.config).execute(
collection,
- archive,
in_progress,
slug,
deposit_id=deposit_id,
+ archive_path=archive,
replace=replace,
)
- elif not archive and metadata:
- r = UpdateMetadataDepositClient(self.config).execute(
+ elif not archive and metadata and swhid is None:
+ r = UpdateMetadataOnPartialDepositClient(self.config).execute(
collection,
- metadata,
in_progress,
slug,
deposit_id=deposit_id,
+ metadata_path=metadata,
replace=replace,
)
+ elif not archive and metadata and swhid is not None:
+ r = UpdateMetadataOnDoneDepositClient(self.config).execute(
+ collection,
+ in_progress,
+ slug,
+ deposit_id=deposit_id,
+ metadata_path=metadata,
+ swhid=swhid,
+ )
else:
r = UpdateMultipartDepositClient(self.config).execute(
collection,
- archive,
- metadata,
in_progress,
slug,
deposit_id=deposit_id,
+ archive_path=archive,
+ metadata_path=metadata,
replace=replace,
)
if "error" in r:
return r
return self.deposit_status(collection, deposit_id)
diff --git a/swh/deposit/config.py b/swh/deposit/config.py
index e67dd030..ec1e0248 100644
--- a/swh/deposit/config.py
+++ b/swh/deposit/config.py
@@ -1,105 +1,103 @@
# Copyright (C) 2017-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import os
from typing import Any, Dict
from swh.core import config
from swh.deposit import __version__
from swh.scheduler import get_scheduler
from swh.scheduler.interface import SchedulerInterface
# IRIs (Internationalized Resource identifier) sword 2.0 specified
EDIT_SE_IRI = "edit_se_iri"
EM_IRI = "em_iri"
CONT_FILE_IRI = "cont_file_iri"
SD_IRI = "servicedocument"
COL_IRI = "upload"
STATE_IRI = "state_iri"
PRIVATE_GET_RAW_CONTENT = "private-download"
PRIVATE_CHECK_DEPOSIT = "check-deposit"
PRIVATE_PUT_DEPOSIT = "private-update"
PRIVATE_GET_DEPOSIT_METADATA = "private-read"
PRIVATE_LIST_DEPOSITS = "private-deposit-list"
ARCHIVE_KEY = "archive"
METADATA_KEY = "metadata"
RAW_METADATA_KEY = "raw-metadata"
ARCHIVE_TYPE = "archive"
METADATA_TYPE = "metadata"
AUTHORIZED_PLATFORMS = ["development", "production", "testing"]
DEPOSIT_STATUS_REJECTED = "rejected"
DEPOSIT_STATUS_PARTIAL = "partial"
DEPOSIT_STATUS_DEPOSITED = "deposited"
DEPOSIT_STATUS_VERIFIED = "verified"
DEPOSIT_STATUS_LOAD_SUCCESS = "done"
DEPOSIT_STATUS_LOAD_FAILURE = "failed"
# Revision author for deposit
SWH_PERSON = {
"name": "Software Heritage",
"fullname": "Software Heritage",
"email": "robot@softwareheritage.org",
}
DEFAULT_CONFIG = {
"max_upload_size": 209715200,
"checks": True,
}
def setup_django_for(platform=None, config_file=None):
"""Setup function for command line tools (swh.deposit.create_user) to
initialize the needed db access.
Note:
Do not import any django related module prior to this function
call. Otherwise, this will raise an
django.core.exceptions.ImproperlyConfigured error message.
Args:
platform (str): the platform the scheduling is running
config_file (str): Extra configuration file (typically for the
production platform)
Raises:
ValueError in case of wrong platform inputs.
"""
if platform is not None:
if platform not in AUTHORIZED_PLATFORMS:
raise ValueError("Platform should be one of %s" % AUTHORIZED_PLATFORMS)
if "DJANGO_SETTINGS_MODULE" not in os.environ:
os.environ["DJANGO_SETTINGS_MODULE"] = "swh.deposit.settings.%s" % platform
if config_file:
os.environ.setdefault("SWH_CONFIG_FILENAME", config_file)
import django
django.setup()
class APIConfig:
"""API Configuration centralized class. This loads explicitly the configuration file out
of the SWH_CONFIG_FILENAME environment variable.
"""
def __init__(self):
- config_file = os.environ["SWH_CONFIG_FILENAME"]
- conf = config.read_raw_config(config.config_basepath(config_file))
- self.config: Dict[str, Any] = config.merge_configs(DEFAULT_CONFIG, conf)
+ self.config: Dict[str, Any] = config.load_from_envvar(DEFAULT_CONFIG)
self.scheduler: SchedulerInterface = get_scheduler(**self.config["scheduler"])
self.tool = {
"name": "swh-deposit",
"version": __version__,
"configuration": {"sword_version": "2"},
}
diff --git a/swh/deposit/loader/checker.py b/swh/deposit/loader/checker.py
index 5e239083..9f491033 100644
--- a/swh/deposit/loader/checker.py
+++ b/swh/deposit/loader/checker.py
@@ -1,42 +1,38 @@
# Copyright (C) 2017-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import logging
-import os
from typing import Any, Dict
from swh.core import config
from swh.deposit.client import PrivateApiDepositClient
logger = logging.getLogger(__name__)
class DepositChecker:
"""Deposit checker implementation.
Trigger deposit's checks through the private api.
"""
def __init__(self):
- config_file = os.environ["SWH_CONFIG_FILENAME"]
- self.config: Dict[str, Any] = config.read_raw_config(
- config.config_basepath(config_file)
- )
+ self.config: Dict[str, Any] = config.load_from_envvar()
self.client = PrivateApiDepositClient(config=self.config["deposit"])
def check(self, collection: str, deposit_id: str) -> Dict[str, str]:
status = None
deposit_check_url = f"/{collection}/{deposit_id}/check/"
logger.debug("deposit-check-url: %s", deposit_check_url)
try:
r = self.client.check(deposit_check_url)
logger.debug("Check result: %s", r)
status = "eventful" if r == "verified" else "failed"
except Exception:
logger.exception("Failure during check on '%s'", deposit_check_url)
status = "failed"
logger.debug("Check status: %s", status)
return {"status": status}
diff --git a/swh/deposit/tests/api/test_checks.py b/swh/deposit/tests/api/test_checks.py
index 58223541..c55562c4 100644
--- a/swh/deposit/tests/api/test_checks.py
+++ b/swh/deposit/tests/api/test_checks.py
@@ -1,78 +1,66 @@
# Copyright (C) 2017-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
+import pytest
+
from swh.deposit.api.checks import check_metadata
-def test_api_checks_check_metadata_ok(swh_checks_deposit):
- actual_check, detail = check_metadata(
+@pytest.mark.parametrize(
+ "metadata_ok",
+ [
{
"url": "something",
"external_identifier": "something-else",
"name": "foo",
"author": "someone",
- }
- )
-
- assert actual_check is True
- assert detail is None
-
-
-def test_api_checks_check_metadata_ok2(swh_checks_deposit):
- actual_check, detail = check_metadata(
+ },
{
- "url": "something",
- "external_identifier": "something-else",
+ "url": "some url",
+ "external_identifier": "some id",
"title": "bar",
- "author": "someone",
- }
- )
-
+ "author": "no one",
+ },
+ ],
+)
+def test_api_checks_check_metadata_ok(metadata_ok, swh_checks_deposit):
+ actual_check, detail = check_metadata(metadata_ok)
assert actual_check is True
assert detail is None
-def test_api_checks_check_metadata_ko(swh_checks_deposit):
- """Missing optional field should be caught
-
- """
- actual_check, error_detail = check_metadata(
- {
- "url": "something",
- "external_identifier": "something-else",
- "author": "someone",
- }
- )
-
- expected_error = {
- "metadata": [
+@pytest.mark.parametrize(
+ "metadata_ko,expected_summary",
+ [
+ (
+ {
+ "url": "something",
+ "external_identifier": "something-else",
+ "author": "someone",
+ },
{
"summary": "Mandatory alternate fields are missing",
"fields": ["name or title"],
- }
- ]
- }
- assert actual_check is False
- assert error_detail == expected_error
-
-
-def test_api_checks_check_metadata_ko2(swh_checks_deposit):
- """Missing mandatory fields should be caught
+ },
+ ),
+ (
+ {
+ "url": "something",
+ "external_identifier": "something-else",
+ "title": "foobar",
+ },
+ {"summary": "Mandatory fields are missing", "fields": ["author"],},
+ ),
+ ],
+)
+def test_api_checks_check_metadata_ko(
+ metadata_ko, expected_summary, swh_checks_deposit
+):
+ """Missing optional field should be caught
"""
- actual_check, error_detail = check_metadata(
- {
- "url": "something",
- "external_identifier": "something-else",
- "title": "foobar",
- }
- )
-
- expected_error = {
- "metadata": [{"summary": "Mandatory fields are missing", "fields": ["author"],}]
- }
-
+ actual_check, error_detail = check_metadata(metadata_ko)
assert actual_check is False
- assert error_detail == expected_error
+ assert error_detail == {"metadata": [expected_summary]}
diff --git a/swh/deposit/tests/api/test_deposit_content.py b/swh/deposit/tests/api/test_deposit_content.py
new file mode 100644
index 00000000..5a77a8be
--- /dev/null
+++ b/swh/deposit/tests/api/test_deposit_content.py
@@ -0,0 +1,49 @@
+# Copyright (C) 2020 The Software Heritage developers
+# See the AUTHORS file at the top-level directory of this distribution
+# License: GNU General Public License version 3, or any later version
+# See top-level LICENSE file for more information
+
+from django.urls import reverse
+from rest_framework import status
+
+from swh.deposit.config import CONT_FILE_IRI
+from swh.deposit.models import DEPOSIT_STATUS_DETAIL
+from swh.deposit.parsers import parse_xml
+
+
+def test_api_deposit_content_nominal(
+ client, complete_deposit, partial_deposit_only_metadata
+):
+ """Retrieve information on deposit should return 200 response
+
+ """
+
+ for deposit in [complete_deposit, partial_deposit_only_metadata]:
+ expected_deposit = {
+ "deposit_id": str(deposit.id),
+ "deposit_status": deposit.status,
+ "deposit_status_detail": DEPOSIT_STATUS_DETAIL[deposit.status],
+ }
+
+ url = reverse(CONT_FILE_IRI, args=[deposit.collection.name, deposit.id])
+ response = client.get(url)
+ assert response.status_code == status.HTTP_200_OK
+ actual_deposit = dict(parse_xml(response.content))
+ expected_deposit["sword:request"] = actual_deposit["sword:request"]
+ assert actual_deposit == expected_deposit
+
+
+def test_api_deposit_content_unknown(client, complete_deposit, deposit_collection):
+ """Retrieve information on unknown deposit or collection should return 404
+
+ """
+ unknown_deposit_id = 999
+ unknown_collection = "unknown"
+ for collection, deposit_id in [
+ (deposit_collection.name, unknown_deposit_id),
+ (unknown_collection, complete_deposit.id),
+ (complete_deposit.collection.name, complete_deposit.id + 10),
+ ]:
+ url = reverse(CONT_FILE_IRI, args=[collection, deposit_id])
+ response = client.get(url)
+ assert response.status_code == status.HTTP_404_NOT_FOUND
diff --git a/swh/deposit/tests/cli/test_client.py b/swh/deposit/tests/cli/test_client.py
index 2e793fa4..b5064e1d 100644
--- a/swh/deposit/tests/cli/test_client.py
+++ b/swh/deposit/tests/cli/test_client.py
@@ -1,463 +1,723 @@
# Copyright (C) 2019-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
+import ast
import contextlib
+import json
import logging
import os
-import re
from unittest.mock import MagicMock
from click.testing import CliRunner
import pytest
+import yaml
from swh.deposit.cli import deposit as cli
from swh.deposit.cli.client import InputError, _client, _collection, _url, generate_slug
from swh.deposit.client import MaintenanceError, PublicApiDepositClient
+from swh.deposit.parsers import parse_xml
from ..conftest import TEST_USER
-EXAMPLE_SERVICE_DOCUMENT = {
- "service": {"workspace": {"collection": {"sword:name": "softcol",}}}
-}
+
+@pytest.fixture
+def deposit_config():
+ return {
+ "url": "https://deposit.swh.test/1",
+ "auth": {"username": "test", "password": "test",},
+ }
@pytest.fixture
def datadir(request):
"""Override default datadir to target main test datadir"""
return os.path.join(os.path.dirname(str(request.fspath)), "../data")
@pytest.fixture
def slug():
return generate_slug()
@pytest.fixture
-def client_mock(mocker, slug):
- """A successful deposit client with hard-coded default values
+def patched_tmp_path(tmp_path, mocker):
+ mocker.patch(
+ "tempfile.TemporaryDirectory",
+ return_value=contextlib.nullcontext(str(tmp_path)),
+ )
+ return tmp_path
- """
- mocker.patch("swh.deposit.cli.client.generate_slug", return_value=slug)
- mock_client = MagicMock()
- mocker.patch("swh.deposit.cli.client._client", return_value=mock_client)
- mock_client.service_document.return_value = EXAMPLE_SERVICE_DOCUMENT
- mock_client.deposit_create.return_value = '{"foo": "bar"}'
- return mock_client
+
+@pytest.fixture
+def cli_runner():
+ return CliRunner()
@pytest.fixture
def client_mock_api_down(mocker, slug):
"""A mock client whose connection with api fails due to maintenance issue
"""
- mocker.patch("swh.deposit.cli.client.generate_slug", return_value=slug)
mock_client = MagicMock()
mocker.patch("swh.deposit.cli.client._client", return_value=mock_client)
mock_client.service_document.side_effect = MaintenanceError(
"Database backend maintenance: Temporarily unavailable, try again later."
)
return mock_client
-def test_url():
+def test_cli_url():
assert _url("http://deposit") == "http://deposit/1"
assert _url("https://other/1") == "https://other/1"
-def test_client():
+def test_cli_client():
client = _client("http://deposit", "user", "pass")
assert isinstance(client, PublicApiDepositClient)
-def test_collection_error():
+def test_cli_collection_error():
mock_client = MagicMock()
mock_client.service_document.return_value = {"error": "something went wrong"}
with pytest.raises(InputError) as e:
_collection(mock_client)
assert "Service document retrieval: something went wrong" == str(e.value)
-def test_collection_ok():
- mock_client = MagicMock()
- mock_client.service_document.return_value = EXAMPLE_SERVICE_DOCUMENT
- collection_name = _collection(mock_client)
-
- assert collection_name == "softcol"
+def test_cli_collection_ok(deposit_config, requests_mock_datadir):
+ client = PublicApiDepositClient(deposit_config)
+ collection_name = _collection(client)
+ assert collection_name == "test"
-def test_collection_ko_because_downtime():
+def test_cli_collection_ko_because_downtime():
mock_client = MagicMock()
mock_client.service_document.side_effect = MaintenanceError("downtime")
with pytest.raises(MaintenanceError, match="downtime"):
_collection(mock_client)
-def test_deposit_with_server_down_for_maintenance(
- sample_archive, mocker, caplog, client_mock_api_down, slug, tmp_path
+def test_cli_deposit_with_server_down_for_maintenance(
+ sample_archive, caplog, client_mock_api_down, slug, patched_tmp_path, cli_runner
):
""" Deposit failure due to maintenance down time should be explicit
"""
- runner = CliRunner()
- result = runner.invoke(
+ result = cli_runner.invoke(
cli,
[
"upload",
"--url",
- "mock://deposit.swh/1",
+ "https://deposit.swh.test/1",
"--username",
TEST_USER["username"],
"--password",
TEST_USER["password"],
"--name",
"test-project",
"--archive",
sample_archive["path"],
"--author",
"Jane Doe",
],
)
assert result.exit_code == 1, result.output
assert result.output == ""
- assert caplog.record_tuples == [
- (
- "swh.deposit.cli.client",
- logging.ERROR,
- "Database backend maintenance: Temporarily unavailable, try again later.",
- )
- ]
+ down_for_maintenance_log_record = (
+ "swh.deposit.cli.client",
+ logging.ERROR,
+ "Database backend maintenance: Temporarily unavailable, try again later.",
+ )
+ assert down_for_maintenance_log_record in caplog.record_tuples
client_mock_api_down.service_document.assert_called_once_with()
-def test_single_minimal_deposit(
- sample_archive, mocker, caplog, client_mock, slug, tmp_path
+def test_cli_single_minimal_deposit(
+ sample_archive, slug, patched_tmp_path, requests_mock_datadir, cli_runner
):
- """ from:
+ """ This ensure a single deposit upload through the cli is fine, cf.
https://docs.softwareheritage.org/devel/swh-deposit/getting-started.html#single-deposit
""" # noqa
- metadata_path = os.path.join(tmp_path, "metadata.xml")
- mocker.patch(
- "tempfile.TemporaryDirectory",
- return_value=contextlib.nullcontext(str(tmp_path)),
- )
-
- runner = CliRunner()
- result = runner.invoke(
+ metadata_path = os.path.join(patched_tmp_path, "metadata.xml")
+ result = cli_runner.invoke(
cli,
[
"upload",
"--url",
- "mock://deposit.swh/1",
+ "https://deposit.swh.test/1",
"--username",
TEST_USER["username"],
"--password",
TEST_USER["password"],
"--name",
"test-project",
"--archive",
sample_archive["path"],
"--author",
"Jane Doe",
+ "--slug",
+ slug,
+ "--format",
+ "json",
],
)
assert result.exit_code == 0, result.output
- assert result.output == ""
- assert caplog.record_tuples == [
- ("swh.deposit.cli.client", logging.INFO, '{"foo": "bar"}'),
- ]
-
- client_mock.deposit_create.assert_called_once_with(
- archive=sample_archive["path"],
- collection="softcol",
- in_progress=False,
- metadata=metadata_path,
- slug=slug,
- )
+ assert json.loads(result.output) == {
+ "deposit_id": "615",
+ "deposit_status": "partial",
+ "deposit_status_detail": None,
+ "deposit_date": "Oct. 8, 2020, 4:57 p.m.",
+ }
with open(metadata_path) as fd:
assert (
fd.read()
== f"""\
<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom" \
xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0">
\t<codemeta:name>test-project</codemeta:name>
\t<codemeta:identifier>{slug}</codemeta:identifier>
\t<codemeta:author>
\t\t<codemeta:name>Jane Doe</codemeta:name>
\t</codemeta:author>
</entry>"""
)
-def test_metadata_validation(sample_archive, mocker, caplog, tmp_path):
- """ from:
- https://docs.softwareheritage.org/devel/swh-deposit/getting-started.html#single-deposit
- """ # noqa
- slug = generate_slug()
- mocker.patch("swh.deposit.cli.client.generate_slug", return_value=slug)
- mock_client = MagicMock()
- mocker.patch("swh.deposit.cli.client._client", return_value=mock_client)
- mock_client.service_document.return_value = EXAMPLE_SERVICE_DOCUMENT
- mock_client.deposit_create.return_value = '{"foo": "bar"}'
+def test_cli_validation_metadata(
+ sample_archive, caplog, patched_tmp_path, cli_runner, slug
+):
+ """Multiple metadata flags scenario (missing, conflicts) properly fails the calls
- metadata_path = os.path.join(tmp_path, "metadata.xml")
- mocker.patch(
- "tempfile.TemporaryDirectory",
- return_value=contextlib.nullcontext(str(tmp_path)),
- )
+ """
+
+ metadata_path = os.path.join(patched_tmp_path, "metadata.xml")
with open(metadata_path, "a"):
pass # creates the file
- runner = CliRunner()
+ for flag_title_or_name, author_or_name in [
+ ("--author", "no one"),
+ ("--name", "test-project"),
+ ]:
+ # Test missing author then missing name
+ result = cli_runner.invoke(
+ cli,
+ [
+ "upload",
+ "--url",
+ "https://deposit.swh.test/1",
+ "--username",
+ TEST_USER["username"],
+ "--password",
+ TEST_USER["password"],
+ "--archive",
+ sample_archive["path"],
+ "--slug",
+ slug,
+ flag_title_or_name,
+ author_or_name,
+ ],
+ )
+
+ assert result.exit_code == 1, f"unexpected result: {result.output}"
+ assert result.output == ""
+ expected_error_log_record = (
+ "swh.deposit.cli.client",
+ logging.ERROR,
+ (
+ "Problem during parsing options: "
+ "For metadata deposit request, either a metadata file with "
+ "--metadata or both --author and --name must be provided. "
+ "If this is an archive deposit request, none is required."
+ ),
+ )
+ assert expected_error_log_record in caplog.record_tuples
+
+ # Clear mocking state
+ caplog.clear()
+
+ # incompatible flags: Test both --metadata and --author, then --metadata and
+ # --name
+ result = cli_runner.invoke(
+ cli,
+ [
+ "upload",
+ "--url",
+ "https://deposit.swh.test/1",
+ "--username",
+ TEST_USER["username"],
+ "--password",
+ TEST_USER["password"],
+ "--name",
+ "test-project",
+ "--deposit-id",
+ 666,
+ "--archive",
+ sample_archive["path"],
+ "--slug",
+ slug,
+ ],
+ )
+ assert result.exit_code == 1, f"unexpected result: {result.output}"
+ assert result.output == ""
+ expected_error_log_record = (
+ "swh.deposit.cli.client",
+ logging.ERROR,
+ (
+ "Problem during parsing options: "
+ "For metadata deposit request, either a metadata file with "
+ "--metadata or both --author and --name must be provided."
+ ),
+ )
+ assert expected_error_log_record in caplog.record_tuples
+
+ # Clear mocking state
+ caplog.clear()
+
+ # incompatible flags check (Test both --metadata and --author,
+ # then --metadata and --name)
+ result = cli_runner.invoke(
+ cli,
+ [
+ "upload",
+ "--url",
+ "https://deposit.swh.test/1",
+ "--username",
+ TEST_USER["username"],
+ "--password",
+ TEST_USER["password"],
+ "--archive",
+ sample_archive["path"],
+ "--metadata",
+ metadata_path,
+ "--author",
+ "Jane Doe",
+ "--slug",
+ slug,
+ ],
+ )
+
+ assert result.exit_code == 1, result.output
+ assert result.output == ""
+ expected_error_log_record = (
+ "swh.deposit.cli.client",
+ logging.ERROR,
+ (
+ "Problem during parsing options: "
+ "Using --metadata flag is incompatible with both "
+ "--author and --name (Those are used to generate one metadata file)."
+ ),
+ )
+ assert expected_error_log_record in caplog.record_tuples
+ caplog.clear()
+
- # Test missing author
- result = runner.invoke(
+def test_cli_validation_no_actionable_command(caplog, cli_runner):
+ """Multiple metadata flags scenario (missing, conflicts) properly fails the calls
+
+ """
+ # no actionable command
+ result = cli_runner.invoke(
cli,
[
"upload",
"--url",
- "mock://deposit.swh/1",
+ "https://deposit.swh.test/1",
"--username",
TEST_USER["username"],
"--password",
TEST_USER["password"],
- "--name",
- "test-project",
- "--archive",
- sample_archive["path"],
+ "--partial",
],
)
assert result.exit_code == 1, result.output
assert result.output == ""
- assert len(caplog.record_tuples) == 1
- (_logger, level, message) = caplog.record_tuples[0]
- assert level == logging.ERROR
- assert " --author " in message
+ expected_error_log_record = (
+ "swh.deposit.cli.client",
+ logging.ERROR,
+ (
+ "Problem during parsing options: "
+ "Please provide an actionable command. See --help for more information"
+ ),
+ )
+ assert expected_error_log_record in caplog.record_tuples
- # Clear mocking state
- caplog.clear()
- mock_client.reset_mock()
- # Test missing name
- result = runner.invoke(
+def test_cli_validation_missing_metadata_flag(caplog, cli_runner):
+ """--metadata-deposit requires --metadata (or --name and --author) otherwise fails
+
+ """
+ # --metadata-deposit without --metadata flag fails
+ result = cli_runner.invoke(
cli,
[
"upload",
"--url",
- "mock://deposit.swh/1",
+ "https://deposit.swh.test/1",
"--username",
TEST_USER["username"],
"--password",
TEST_USER["password"],
- "--archive",
- sample_archive["path"],
- "--author",
- "Jane Doe",
+ "--metadata-deposit", # should fail because missing --metadata flag
],
)
assert result.exit_code == 1, result.output
assert result.output == ""
- assert len(caplog.record_tuples) == 1
- (_logger, level, message) = caplog.record_tuples[0]
- assert level == logging.ERROR
- assert " --name " in message
+ expected_error_log_record = (
+ "swh.deposit.cli.client",
+ logging.ERROR,
+ (
+ "Problem during parsing options: "
+ "Metadata deposit must be provided for metadata "
+ "deposit, either a filepath with --metadata or --name and --author"
+ ),
+ )
+ assert expected_error_log_record in caplog.record_tuples
+
- # Clear mocking state
- caplog.clear()
- mock_client.reset_mock()
+def test_cli_validation_replace_with_no_deposit_id_fails(
+ sample_archive, caplog, patched_tmp_path, requests_mock_datadir, datadir, cli_runner
+):
+ """--replace flags require --deposit-id otherwise fails
+
+ """
+ metadata_path = os.path.join(datadir, "atom", "entry-data-deposit-binary.xml")
+
+ for flags in [
+ ["--replace"],
+ ["--replace", "--metadata-deposit", "--archive-deposit"],
+ ]:
+ result = cli_runner.invoke(
+ cli,
+ [
+ "upload",
+ "--url",
+ "https://deposit.swh.test/1",
+ "--username",
+ TEST_USER["username"],
+ "--password",
+ TEST_USER["password"],
+ "--metadata",
+ metadata_path,
+ "--archive",
+ sample_archive["path"],
+ ]
+ + flags,
+ )
- # Test both --metadata and --author
- result = runner.invoke(
+ assert result.exit_code == 1, result.output
+ assert result.output == ""
+ expected_error_log_record = (
+ "swh.deposit.cli.client",
+ logging.ERROR,
+ (
+ "Problem during parsing options: "
+ "To update an existing deposit, you must provide its id"
+ ),
+ )
+ assert expected_error_log_record in caplog.record_tuples
+
+
+def test_cli_single_deposit_slug_generation(
+ sample_archive, patched_tmp_path, requests_mock_datadir, cli_runner
+):
+ """Single deposit scenario without providing the slug, the slug is generated nonetheless
+ https://docs.softwareheritage.org/devel/swh-deposit/getting-started.html#single-deposit
+ """ # noqa
+ metadata_path = os.path.join(patched_tmp_path, "metadata.xml")
+ result = cli_runner.invoke(
cli,
[
"upload",
"--url",
- "mock://deposit.swh/1",
+ "https://deposit.swh.test/1",
"--username",
TEST_USER["username"],
"--password",
TEST_USER["password"],
+ "--name",
+ "test-project",
"--archive",
sample_archive["path"],
- "--metadata",
- metadata_path,
"--author",
"Jane Doe",
+ "--format",
+ "json",
],
)
+ assert result.exit_code == 0, result.output
+ assert json.loads(result.output) == {
+ "deposit_id": "615",
+ "deposit_status": "partial",
+ "deposit_status_detail": None,
+ "deposit_date": "Oct. 8, 2020, 4:57 p.m.",
+ }
- assert result.exit_code == 1, result.output
- assert result.output == ""
- assert len(caplog.record_tuples) == 1
- (_logger, level, message) = caplog.record_tuples[0]
- assert level == logging.ERROR
- assert re.search("--metadata.*is incompatible with", message)
-
- # Clear mocking state
- caplog.clear()
- mock_client.reset_mock()
+ with open(metadata_path) as fd:
+ metadata_xml = fd.read()
+ actual_metadata = parse_xml(metadata_xml)
+ assert actual_metadata["codemeta:identifier"] is not None
-def test_single_deposit_slug_generation(
- sample_archive, mocker, caplog, tmp_path, client_mock
+def test_cli_multisteps_deposit(
+ sample_archive, datadir, slug, requests_mock_datadir, cli_runner
):
- """ from:
- https://docs.softwareheritage.org/devel/swh-deposit/getting-started.html#single-deposit
+ """ First deposit a partial deposit (no metadata, only archive), then update the metadata part.
+ https://docs.softwareheritage.org/devel/swh-deposit/getting-started.html#multisteps-deposit
""" # noqa
- slug = "my-slug"
- collection = "my-collection"
+ api_url = "https://deposit.test.metadata/1"
+ deposit_id = 666
- metadata_path = os.path.join(tmp_path, "metadata.xml")
- mocker.patch(
- "tempfile.TemporaryDirectory",
- return_value=contextlib.nullcontext(str(tmp_path)),
+ # Create a partial deposit with only 1 archive
+ result = cli_runner.invoke(
+ cli,
+ [
+ "upload",
+ "--url",
+ api_url,
+ "--username",
+ TEST_USER["username"],
+ "--password",
+ TEST_USER["password"],
+ "--archive",
+ sample_archive["path"],
+ "--partial",
+ "--slug",
+ slug,
+ "--format",
+ "json",
+ ],
)
- runner = CliRunner()
- result = runner.invoke(
+ assert result.exit_code == 0, f"unexpected output: {result.output}"
+ actual_deposit = json.loads(result.output)
+ assert actual_deposit == {
+ "deposit_id": str(deposit_id),
+ "deposit_status": "partial",
+ "deposit_status_detail": None,
+ "deposit_date": "Oct. 8, 2020, 4:57 p.m.",
+ }
+
+ # Update the partial deposit with only 1 archive
+ result = cli_runner.invoke(
cli,
[
"upload",
"--url",
- "mock://deposit.swh/1",
+ api_url,
"--username",
TEST_USER["username"],
"--password",
TEST_USER["password"],
- "--name",
- "test-project",
"--archive",
sample_archive["path"],
+ "--deposit-id",
+ deposit_id,
+ "--partial", # in-progress: True, because remains the metadata to upload
"--slug",
slug,
- "--collection",
- collection,
- "--author",
- "Jane Doe",
+ "--format",
+ "json",
],
)
+ assert result.exit_code == 0, f"unexpected output: {result.output}"
+ assert result.output is not None
+ actual_deposit = json.loads(result.output)
+ # deposit update scenario actually returns a deposit status dict
+ assert actual_deposit["deposit_id"] == str(deposit_id)
+ assert actual_deposit["deposit_status"] == "partial"
+
+ # Update the partial deposit with only some metadata (and then finalize it)
+ # https://docs.softwareheritage.org/devel/swh-deposit/getting-started.html#add-content-or-metadata-to-the-deposit
+ metadata_path = os.path.join(datadir, "atom", "entry-data-deposit-binary.xml")
- assert result.exit_code == 0, result.output
- assert result.output == ""
- assert caplog.record_tuples == [
- ("swh.deposit.cli.client", logging.INFO, '{"foo": "bar"}'),
- ]
-
- client_mock.deposit_create.assert_called_once_with(
- archive=sample_archive["path"],
- collection=collection,
- in_progress=False,
- metadata=metadata_path,
- slug=slug,
+ # Update deposit with metadata
+ result = cli_runner.invoke(
+ cli,
+ [
+ "upload",
+ "--url",
+ api_url,
+ "--username",
+ TEST_USER["username"],
+ "--password",
+ TEST_USER["password"],
+ "--metadata",
+ metadata_path,
+ "--deposit-id",
+ deposit_id,
+ "--slug",
+ slug,
+ "--format",
+ "json",
+ ], # this time, ^ we no longer flag it to partial, so the status changes to
+ # in-progress false
)
- with open(metadata_path) as fd:
- assert (
- fd.read()
- == """\
-<?xml version="1.0" encoding="utf-8"?>
-<entry xmlns="http://www.w3.org/2005/Atom" \
-xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0">
-\t<codemeta:name>test-project</codemeta:name>
-\t<codemeta:identifier>my-slug</codemeta:identifier>
-\t<codemeta:author>
-\t\t<codemeta:name>Jane Doe</codemeta:name>
-\t</codemeta:author>
-</entry>"""
- )
-
-
-def test_multisteps_deposit(
- sample_archive, atom_dataset, mocker, caplog, datadir, client_mock, slug
+ assert result.exit_code == 0, f"unexpected output: {result.output}"
+ assert result.output is not None
+ actual_deposit = json.loads(result.output)
+ # deposit update scenario actually returns a deposit status dict
+ assert actual_deposit["deposit_id"] == str(deposit_id)
+ # FIXME: should be "deposited" but current limitation in the
+ # requests_mock_datadir_visits use, cannot find a way to make it work right now
+ assert actual_deposit["deposit_status"] == "partial"
+
+
+@pytest.mark.parametrize(
+ "output_format,callable_fn",
+ [
+ ("json", json.loads),
+ ("yaml", yaml.safe_load),
+ (
+ "logging",
+ ast.literal_eval,
+ ), # not enough though, the caplog fixture is needed
+ ],
+)
+def test_cli_deposit_status_with_output_format(
+ output_format, callable_fn, datadir, slug, requests_mock_datadir, caplog, cli_runner
):
- """ from:
- https://docs.softwareheritage.org/devel/swh-deposit/getting-started.html#multisteps-deposit
- """ # noqa
- slug = generate_slug()
- mocker.patch("swh.deposit.cli.client.generate_slug", return_value=slug)
+ """Check deposit status cli with all possible output formats (json, yaml, logging).
- # https://docs.softwareheritage.org/devel/swh-deposit/getting-started.html#create-an-incomplete-deposit
- client_mock.deposit_create.return_value = '{"deposit_id": "42"}'
+ """
+ api_url_basename = "deposit.test.status"
+ deposit_id = 1033
+ deposit_status_xml_path = os.path.join(
+ datadir, f"https_{api_url_basename}", f"1_test_{deposit_id}_status"
+ )
+ with open(deposit_status_xml_path, "r") as f:
+ deposit_status_xml = f.read()
+ expected_deposit_status = dict(parse_xml(deposit_status_xml))
- runner = CliRunner()
- result = runner.invoke(
+ result = cli_runner.invoke(
cli,
[
- "upload",
+ "status",
"--url",
- "mock://deposit.swh/1",
+ f"https://{api_url_basename}/1",
"--username",
TEST_USER["username"],
"--password",
TEST_USER["password"],
- "--archive",
- sample_archive["path"],
- "--partial",
+ "--deposit-id",
+ deposit_id,
+ "--format",
+ output_format,
],
)
+ assert result.exit_code == 0, f"unexpected output: {result.output}"
- assert result.exit_code == 0, result.output
- assert result.output == ""
- assert caplog.record_tuples == [
- ("swh.deposit.cli.client", logging.INFO, '{"deposit_id": "42"}'),
- ]
-
- client_mock.deposit_create.assert_called_once_with(
- archive=sample_archive["path"],
- collection="softcol",
- in_progress=True,
- metadata=None,
- slug=slug,
- )
+ if output_format == "logging":
+ assert len(caplog.record_tuples) == 1
+ # format: (<module>, <log-level>, <log-msg>)
+ _, _, result_output = caplog.record_tuples[0]
+ else:
+ result_output = result.output
- # Clear mocking state
- caplog.clear()
- client_mock.reset_mock()
+ actual_deposit = callable_fn(result_output)
+ assert actual_deposit == expected_deposit_status
- # https://docs.softwareheritage.org/devel/swh-deposit/getting-started.html#add-content-or-metadata-to-the-deposit
- metadata_path = os.path.join(datadir, "atom", "entry-data-deposit-binary.xml")
+def test_cli_update_metadata_with_swhid_on_completed_deposit(
+ datadir, requests_mock_datadir, cli_runner
+):
+ """Update new metadata on a completed deposit (status done) is ok
+ """
+ api_url_basename = "deposit.test.updateswhid"
+ deposit_id = 123
+ deposit_status_xml_path = os.path.join(
+ datadir, f"https_{api_url_basename}", f"1_test_{deposit_id}_status"
+ )
+ with open(deposit_status_xml_path, "r") as f:
+ deposit_status_xml = f.read()
+ expected_deposit_status = dict(parse_xml(deposit_status_xml))
- result = runner.invoke(
+ assert expected_deposit_status["deposit_status"] == "done"
+ assert expected_deposit_status["deposit_swh_id"] is not None
+
+ result = cli_runner.invoke(
cli,
[
"upload",
"--url",
- "mock://deposit.swh/1",
+ f"https://{api_url_basename}/1",
"--username",
TEST_USER["username"],
"--password",
TEST_USER["password"],
- "--metadata",
- metadata_path,
+ "--name",
+ "test-project",
+ "--author",
+ "John Doe",
+ "--deposit-id",
+ deposit_id,
+ "--swhid",
+ expected_deposit_status["deposit_swh_id"],
+ "--format",
+ "json",
],
)
-
assert result.exit_code == 0, result.output
- assert result.output == ""
- assert caplog.record_tuples == [
- ("swh.deposit.cli.client", logging.INFO, '{"deposit_id": "42"}'),
- ]
-
- client_mock.deposit_create.assert_called_once_with(
- archive=None,
- collection="softcol",
- in_progress=False,
- metadata=metadata_path,
- slug=slug,
+ actual_deposit_status = json.loads(result.output)
+ assert "error" not in actual_deposit_status
+ assert actual_deposit_status == expected_deposit_status
+
+
+def test_cli_update_metadata_with_swhid_on_other_status_deposit(
+ datadir, requests_mock_datadir, cli_runner
+):
+ """Update new metadata with swhid on other deposit status is not possible
+ """
+ api_url_basename = "deposit.test.updateswhid"
+ deposit_id = 321
+ deposit_status_xml_path = os.path.join(
+ datadir, f"https_{api_url_basename}", f"1_test_{deposit_id}_status"
)
+ with open(deposit_status_xml_path, "r") as f:
+ deposit_status_xml = f.read()
+ expected_deposit_status = dict(parse_xml(deposit_status_xml))
+ assert expected_deposit_status["deposit_status"] != "done"
- # Clear mocking state
- caplog.clear()
- client_mock.reset_mock()
+ result = cli_runner.invoke(
+ cli,
+ [
+ "upload",
+ "--url",
+ f"https://{api_url_basename}/1",
+ "--username",
+ TEST_USER["username"],
+ "--password",
+ TEST_USER["password"],
+ "--name",
+ "test-project",
+ "--author",
+ "John Doe",
+ "--deposit-id",
+ deposit_id,
+ "--swhid",
+ "swh:1:dir:ef04a768181417fbc5eef4243e2507915f24deea",
+ "--format",
+ "json",
+ ],
+ )
+ assert result.exit_code == 0, result.output
+ actual_result = json.loads(result.output)
+ assert "error" in actual_result
+ assert actual_result == {
+ "error": "You can only update metadata on deposit with status 'done'",
+ "detail": "The deposit 321 has status 'partial'",
+ "deposit_status": "partial",
+ "deposit_id": 321,
+ }
diff --git a/swh/deposit/tests/conftest.py b/swh/deposit/tests/conftest.py
index 9717653f..7bbb3f20 100644
--- a/swh/deposit/tests/conftest.py
+++ b/swh/deposit/tests/conftest.py
@@ -1,418 +1,432 @@
# Copyright (C) 2019-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import base64
+from functools import partial
import os
+import re
from typing import Mapping
from django.test.utils import setup_databases # type: ignore
from django.urls import reverse
import psycopg2
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT
import pytest
from rest_framework import status
from rest_framework.test import APIClient
import yaml
from swh.core.config import read
+from swh.core.pytest_plugin import get_response_cb
from swh.deposit.config import (
COL_IRI,
DEPOSIT_STATUS_DEPOSITED,
DEPOSIT_STATUS_LOAD_FAILURE,
DEPOSIT_STATUS_LOAD_SUCCESS,
DEPOSIT_STATUS_PARTIAL,
DEPOSIT_STATUS_REJECTED,
DEPOSIT_STATUS_VERIFIED,
EDIT_SE_IRI,
setup_django_for,
)
from swh.deposit.parsers import parse_xml
from swh.deposit.tests.common import create_arborescence_archive
from swh.model.identifiers import DIRECTORY, REVISION, SNAPSHOT, swhid
from swh.scheduler import get_scheduler
# mypy is asked to ignore the import statement above because setup_databases
# is not part of the d.t.utils.__all__ variable.
TEST_USER = {
"username": "test",
"password": "password",
"email": "test@example.org",
"provider_url": "https://hal-test.archives-ouvertes.fr/",
"domain": "archives-ouvertes.fr/",
"collection": {"name": "test"},
}
def pytest_configure():
setup_django_for("testing")
+@pytest.fixture
+def requests_mock_datadir(datadir, requests_mock_datadir):
+ """Override default behavior to deal with put/post methods
+
+ """
+ cb = partial(get_response_cb, datadir=datadir)
+ requests_mock_datadir.put(re.compile("https://"), body=cb)
+ requests_mock_datadir.post(re.compile("https://"), body=cb)
+ return requests_mock_datadir
+
+
@pytest.fixture()
def deposit_config(swh_scheduler_config, swh_storage_backend_config):
return {
"max_upload_size": 500,
"extraction_dir": "/tmp/swh-deposit/test/extraction-dir",
"checks": False,
"scheduler": {"cls": "local", "args": swh_scheduler_config,},
"storage_metadata": swh_storage_backend_config,
}
@pytest.fixture()
def deposit_config_path(tmp_path, monkeypatch, deposit_config):
conf_path = os.path.join(tmp_path, "deposit.yml")
with open(conf_path, "w") as f:
f.write(yaml.dump(deposit_config))
monkeypatch.setenv("SWH_CONFIG_FILENAME", conf_path)
return conf_path
@pytest.fixture(autouse=True)
def deposit_autoconfig(deposit_config_path, swh_scheduler_config):
"""Enforce config for deposit classes inherited from APIConfig."""
cfg = read(deposit_config_path)
if "scheduler" in cfg:
# scheduler setup: require the load-deposit tasks (already existing in
# production)
scheduler = get_scheduler(**cfg["scheduler"])
task_type = {
"type": "load-deposit",
"backend_name": "swh.loader.packages.deposit.tasks.LoadDeposit",
"description": "Load deposit task",
}
scheduler.create_task_type(task_type)
@pytest.fixture(scope="session")
def django_db_setup(request, django_db_blocker, postgresql_proc):
from django.conf import settings
settings.DATABASES["default"].update(
{
("ENGINE", "django.db.backends.postgresql"),
("NAME", "tests"),
("USER", postgresql_proc.user), # noqa
("HOST", postgresql_proc.host), # noqa
("PORT", postgresql_proc.port), # noqa
}
)
with django_db_blocker.unblock():
setup_databases(
verbosity=request.config.option.verbose, interactive=False, keepdb=False
)
def execute_sql(sql):
"""Execute sql to postgres db"""
with psycopg2.connect(database="postgres") as conn:
conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
cur = conn.cursor()
cur.execute(sql)
@pytest.fixture(autouse=True, scope="session")
def swh_proxy():
"""Automatically inject this fixture in all tests to ensure no outside
connection takes place.
"""
os.environ["http_proxy"] = "http://localhost:999"
os.environ["https_proxy"] = "http://localhost:999"
def create_deposit_collection(collection_name: str):
"""Create a deposit collection with name collection_name
"""
from swh.deposit.models import DepositCollection
try:
collection = DepositCollection._default_manager.get(name=collection_name)
except DepositCollection.DoesNotExist:
collection = DepositCollection(name=collection_name)
collection.save()
return collection
def deposit_collection_factory(collection_name=TEST_USER["collection"]["name"]):
@pytest.fixture
def _deposit_collection(db, collection_name=collection_name):
return create_deposit_collection(collection_name)
return _deposit_collection
deposit_collection = deposit_collection_factory()
deposit_another_collection = deposit_collection_factory("another-collection")
@pytest.fixture
def deposit_user(db, deposit_collection):
"""Create/Return the test_user "test"
"""
from swh.deposit.models import DepositClient
try:
user = DepositClient._default_manager.get(username=TEST_USER["username"])
except DepositClient.DoesNotExist:
user = DepositClient._default_manager.create_user(
username=TEST_USER["username"],
email=TEST_USER["email"],
password=TEST_USER["password"],
provider_url=TEST_USER["provider_url"],
domain=TEST_USER["domain"],
)
user.collections = [deposit_collection.id]
user.save()
return user
@pytest.fixture
def client():
"""Override pytest-django one which does not work for djangorestframework.
"""
return APIClient() # <- drf's client
@pytest.yield_fixture
def authenticated_client(client, deposit_user):
"""Returned a logged client
"""
_token = "%s:%s" % (deposit_user.username, TEST_USER["password"])
token = base64.b64encode(_token.encode("utf-8"))
authorization = "Basic %s" % token.decode("utf-8")
client.credentials(HTTP_AUTHORIZATION=authorization)
yield client
client.logout()
@pytest.fixture
def sample_archive(tmp_path):
"""Returns a sample archive
"""
tmp_path = str(tmp_path) # pytest version limitation in previous version
archive = create_arborescence_archive(
tmp_path, "archive1", "file1", b"some content in file"
)
return archive
@pytest.fixture
def atom_dataset(datadir) -> Mapping[str, str]:
"""Compute the paths to atom files.
Returns:
Dict of atom name per content (bytes)
"""
atom_path = os.path.join(datadir, "atom")
data = {}
for filename in os.listdir(atom_path):
filepath = os.path.join(atom_path, filename)
with open(filepath, "rb") as f:
raw_content = f.read().decode("utf-8")
# Keep the filename without extension
atom_name = filename.split(".")[0]
data[atom_name] = raw_content
return data
def create_deposit(
authenticated_client,
collection_name: str,
sample_archive,
external_id: str,
deposit_status=DEPOSIT_STATUS_DEPOSITED,
):
"""Create a skeleton shell deposit
"""
url = reverse(COL_IRI, args=[collection_name])
# when
response = authenticated_client.post(
url,
content_type="application/zip", # as zip
data=sample_archive["data"],
# + headers
CONTENT_LENGTH=sample_archive["length"],
HTTP_SLUG=external_id,
HTTP_CONTENT_MD5=sample_archive["md5sum"],
HTTP_PACKAGING="http://purl.org/net/sword/package/SimpleZip",
HTTP_IN_PROGRESS="false",
HTTP_CONTENT_DISPOSITION="attachment; filename=%s" % (sample_archive["name"]),
)
# then
assert response.status_code == status.HTTP_201_CREATED
from swh.deposit.models import Deposit
deposit = Deposit._default_manager.get(external_id=external_id)
if deposit.status != deposit_status:
deposit.status = deposit_status
deposit.save()
assert deposit.status == deposit_status
return deposit
def create_binary_deposit(
authenticated_client,
collection_name: str,
sample_archive,
external_id: str,
deposit_status: str = DEPOSIT_STATUS_DEPOSITED,
atom_dataset: Mapping[str, bytes] = {},
):
"""Create a deposit with both metadata and archive set. Then alters its status
to `deposit_status`.
"""
deposit = create_deposit(
authenticated_client,
collection_name,
sample_archive,
external_id=external_id,
deposit_status=DEPOSIT_STATUS_PARTIAL,
)
response = authenticated_client.post(
reverse(EDIT_SE_IRI, args=[collection_name, deposit.id]),
content_type="application/atom+xml;type=entry",
data=atom_dataset["entry-data0"] % deposit.external_id.encode("utf-8"),
HTTP_SLUG=deposit.external_id,
HTTP_IN_PROGRESS="true",
)
assert response.status_code == status.HTTP_201_CREATED
assert deposit.status == DEPOSIT_STATUS_PARTIAL
from swh.deposit.models import Deposit
deposit = Deposit._default_manager.get(pk=deposit.id)
if deposit.status != deposit_status:
deposit.status = deposit_status
deposit.save()
assert deposit.status == deposit_status
return deposit
def deposit_factory(deposit_status=DEPOSIT_STATUS_DEPOSITED):
"""Build deposit with a specific status
"""
@pytest.fixture()
def _deposit(
sample_archive,
deposit_collection,
authenticated_client,
deposit_status=deposit_status,
):
external_id = "external-id-%s" % deposit_status
return create_deposit(
authenticated_client,
deposit_collection.name,
sample_archive,
external_id=external_id,
deposit_status=deposit_status,
)
return _deposit
deposited_deposit = deposit_factory()
rejected_deposit = deposit_factory(deposit_status=DEPOSIT_STATUS_REJECTED)
partial_deposit = deposit_factory(deposit_status=DEPOSIT_STATUS_PARTIAL)
verified_deposit = deposit_factory(deposit_status=DEPOSIT_STATUS_VERIFIED)
completed_deposit = deposit_factory(deposit_status=DEPOSIT_STATUS_LOAD_SUCCESS)
failed_deposit = deposit_factory(deposit_status=DEPOSIT_STATUS_LOAD_FAILURE)
@pytest.fixture
def partial_deposit_with_metadata(
sample_archive, deposit_collection, authenticated_client, atom_dataset
):
"""Returns deposit with archive and metadata provided, status 'partial'
"""
return create_binary_deposit(
authenticated_client,
deposit_collection.name,
sample_archive,
external_id="external-id-partial",
deposit_status=DEPOSIT_STATUS_PARTIAL,
atom_dataset=atom_dataset,
)
@pytest.fixture
def partial_deposit_only_metadata(
deposit_collection, authenticated_client, atom_dataset
):
response = authenticated_client.post(
reverse(COL_IRI, args=[deposit_collection.name]),
content_type="application/atom+xml;type=entry",
data=atom_dataset["entry-data1"],
HTTP_SLUG="external-id-partial",
HTTP_IN_PROGRESS=True,
)
assert response.status_code == status.HTTP_201_CREATED
response_content = parse_xml(response.content)
deposit_id = response_content["deposit_id"]
from swh.deposit.models import Deposit
deposit = Deposit._default_manager.get(pk=deposit_id)
assert deposit.status == DEPOSIT_STATUS_PARTIAL
return deposit
@pytest.fixture
def complete_deposit(sample_archive, deposit_collection, authenticated_client):
"""Returns a completed deposit (load success)
"""
deposit = create_deposit(
authenticated_client,
deposit_collection.name,
sample_archive,
external_id="external-id-complete",
deposit_status=DEPOSIT_STATUS_LOAD_SUCCESS,
)
origin = "https://hal.archives-ouvertes.fr/hal-01727745"
directory_id = "42a13fc721c8716ff695d0d62fc851d641f3a12b"
revision_id = "548b3c0a2bb43e1fca191e24b5803ff6b3bc7c10"
snapshot_id = "e5e82d064a9c3df7464223042e0c55d72ccff7f0"
deposit.swhid = swhid(DIRECTORY, directory_id)
deposit.swhid_context = swhid(
DIRECTORY,
directory_id,
metadata={
"origin": origin,
"visit": swhid(SNAPSHOT, snapshot_id),
"anchor": swhid(REVISION, revision_id),
"path": "/",
},
)
deposit.save()
return deposit
@pytest.fixture()
def tmp_path(tmp_path):
return str(tmp_path) # issue with oldstable's pytest version
diff --git a/swh/deposit/tests/data/https_deposit.swh.test/1_servicedocument b/swh/deposit/tests/data/https_deposit.swh.test/1_servicedocument
new file mode 100644
index 00000000..236a522d
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.swh.test/1_servicedocument
@@ -0,0 +1,26 @@
+<?xml version="1.0" ?>
+<service xmlns:dcterms="http://purl.org/dc/terms/"
+ xmlns:sword="http://purl.org/net/sword/terms/"
+ xmlns:atom="http://www.w3.org/2005/Atom"
+ xmlns="http://www.w3.org/2007/app">
+
+ <sword:version>2.0</sword:version>
+ <sword:maxUploadSize>209715200</sword:maxUploadSize>
+
+ <workspace>
+ <atom:title>The Software Heritage (SWH) Archive</atom:title>
+ <collection href="test">
+ <atom:title>test Software Collection</atom:title>
+ <accept>application/zip</accept>
+ <accept>application/x-tar</accept>
+ <sword:collectionPolicy>Collection Policy</sword:collectionPolicy>
+ <dcterms:abstract>Software Heritage Archive</dcterms:abstract>
+ <sword:treatment>Collect, Preserve, Share</sword:treatment>
+ <sword:mediation>false</sword:mediation>
+ <sword:metadataRelevantHeader>false</sword:metadataRelevantHeader>
+ <sword:acceptPackaging>http://purl.org/net/sword/package/SimpleZip</sword:acceptPackaging>
+ <sword:service>https://deposit.swh.test/1/test/</sword:service>
+ <sword:name>test</sword:name>
+ </collection>
+ </workspace>
+</service>
diff --git a/swh/deposit/tests/data/https_deposit.swh.test/1_test b/swh/deposit/tests/data/https_deposit.swh.test/1_test
new file mode 100644
index 00000000..6d6d9532
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.swh.test/1_test
@@ -0,0 +1,19 @@
+<entry xmlns="http://www.w3.org/2005/Atom"
+ xmlns:sword="http://purl.org/net/sword/"
+ xmlns:dcterms="http://purl.org/dc/terms/">
+ <deposit_id>615</deposit_id>
+ <deposit_date>Oct. 8, 2020, 4:57 p.m.</deposit_date>
+ <deposit_archive>None</deposit_archive>
+ <deposit_status>partial</deposit_status>
+
+ <!-- Edit-IRI -->
+ <link rel="edit" href="https://deposit.swh.test/1/test/615/metadata/" />
+ <!-- EM-IRI -->
+ <link rel="edit-media" href="https://deposit.swh.test/1/test/615/media/"/>
+ <!-- SE-IRI -->
+ <link rel="http://purl.org/net/sword/terms/add" href="https://deposit.swh.test/1/test/615/metadata/" />
+ <!-- State-IRI -->
+ <link rel="alternate" href="https://deposit.swh.test/1/test/615/status/" />
+
+ <sword:packaging>http://purl.org/net/sword/package/SimpleZip</sword:packaging>
+</entry>
diff --git a/swh/deposit/tests/data/https_deposit.test.metadata/1_servicedocument b/swh/deposit/tests/data/https_deposit.test.metadata/1_servicedocument
new file mode 100644
index 00000000..033ca6ea
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.test.metadata/1_servicedocument
@@ -0,0 +1,26 @@
+<?xml version="1.0" ?>
+<service xmlns:dcterms="http://purl.org/dc/terms/"
+ xmlns:sword="http://purl.org/net/sword/terms/"
+ xmlns:atom="http://www.w3.org/2005/Atom"
+ xmlns="http://www.w3.org/2007/app">
+
+ <sword:version>2.0</sword:version>
+ <sword:maxUploadSize>209715200</sword:maxUploadSize>
+
+ <workspace>
+ <atom:title>The Software Heritage (SWH) Archive</atom:title>
+ <collection href="test">
+ <atom:title>test Software Collection</atom:title>
+ <accept>application/zip</accept>
+ <accept>application/x-tar</accept>
+ <sword:collectionPolicy>Collection Policy</sword:collectionPolicy>
+ <dcterms:abstract>Software Heritage Archive</dcterms:abstract>
+ <sword:treatment>Collect, Preserve, Share</sword:treatment>
+ <sword:mediation>false</sword:mediation>
+ <sword:metadataRelevantHeader>false</sword:metadataRelevantHeader>
+ <sword:acceptPackaging>http://purl.org/net/sword/package/SimpleZip</sword:acceptPackaging>
+ <sword:service>https://deposit.test.metadata/1/test/</sword:service>
+ <sword:name>test</sword:name>
+ </collection>
+ </workspace>
+</service>
diff --git a/swh/deposit/tests/data/https_deposit.test.metadata/1_test b/swh/deposit/tests/data/https_deposit.test.metadata/1_test
new file mode 100644
index 00000000..9622a83f
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.test.metadata/1_test
@@ -0,0 +1,19 @@
+<entry xmlns="http://www.w3.org/2005/Atom"
+ xmlns:sword="http://purl.org/net/sword/"
+ xmlns:dcterms="http://purl.org/dc/terms/">
+ <deposit_id>666</deposit_id>
+ <deposit_date>Oct. 8, 2020, 4:57 p.m.</deposit_date>
+ <deposit_archive>hardcoded_sample_archive_path</deposit_archive>
+ <deposit_status>partial</deposit_status>
+
+ <!-- Edit-IRI -->
+ <link rel="edit" href="https://deposit.test.metadata/1/test/666/metadata/" />
+ <!-- EM-IRI -->
+ <link rel="edit-media" href="https://deposit.test.metadata/1/test/666/media/"/>
+ <!-- SE-IRI -->
+ <link rel="http://purl.org/net/sword/terms/add" href="https://deposit.test.metadata/1/test/666/metadata/" />
+ <!-- State-IRI -->
+ <link rel="alternate" href="https://deposit.test.metadata/1/test/666/status/" />
+
+ <sword:packaging>http://purl.org/net/sword/package/SimpleZip</sword:packaging>
+</entry>
diff --git a/swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_media b/swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_media
new file mode 100644
index 00000000..6a2c89e3
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_media
@@ -0,0 +1,19 @@
+<entry xmlns="http://www.w3.org/2005/Atom"
+ xmlns:sword="http://purl.org/net/sword/"
+ xmlns:dcterms="http://purl.org/dc/terms/">
+ <deposit_id>666</deposit_id>
+ <deposit_date>Oct. 9, 2020, 8:44 p.m.</deposit_date>
+ <deposit_archive>something</deposit_archive>
+ <deposit_status>deposited</deposit_status>
+
+ <!-- Edit-IRI -->
+ <link rel="edit" href="https://deposit.test.metadata/1/test/666/metadata/" />
+ <!-- EM-IRI -->
+ <link rel="edit-media" href="https://deposit.test.metadata/1/test/666/media/"/>
+ <!-- SE-IRI -->
+ <link rel="http://purl.org/net/sword/terms/add" href="https://deposit.test.metadata/1/test/666/metadata/" />
+ <!-- State-IRI -->
+ <link rel="alternate" href="https://deposit.test.metadata/1/test/666/status/" />
+
+ <sword:packaging>http://purl.org/net/sword/package/SimpleZip</sword:packaging>
+</entry>
diff --git a/swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_metadata b/swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_metadata
new file mode 100644
index 00000000..6a2c89e3
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_metadata
@@ -0,0 +1,19 @@
+<entry xmlns="http://www.w3.org/2005/Atom"
+ xmlns:sword="http://purl.org/net/sword/"
+ xmlns:dcterms="http://purl.org/dc/terms/">
+ <deposit_id>666</deposit_id>
+ <deposit_date>Oct. 9, 2020, 8:44 p.m.</deposit_date>
+ <deposit_archive>something</deposit_archive>
+ <deposit_status>deposited</deposit_status>
+
+ <!-- Edit-IRI -->
+ <link rel="edit" href="https://deposit.test.metadata/1/test/666/metadata/" />
+ <!-- EM-IRI -->
+ <link rel="edit-media" href="https://deposit.test.metadata/1/test/666/media/"/>
+ <!-- SE-IRI -->
+ <link rel="http://purl.org/net/sword/terms/add" href="https://deposit.test.metadata/1/test/666/metadata/" />
+ <!-- State-IRI -->
+ <link rel="alternate" href="https://deposit.test.metadata/1/test/666/status/" />
+
+ <sword:packaging>http://purl.org/net/sword/package/SimpleZip</sword:packaging>
+</entry>
diff --git a/swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_status b/swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_status
new file mode 100644
index 00000000..0af5ba9e
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.test.metadata/1_test_666_status
@@ -0,0 +1,8 @@
+<entry xmlns="http://www.w3.org/2005/Atom"
+ xmlns:sword="http://purl.org/net/sword/"
+ xmlns:dcterms="http://purl.org/dc/terms/">
+ <deposit_id>666</deposit_id>
+ <deposit_status>partial</deposit_status>
+ <deposit_status_detail>Deposit is partially received. To finalize it, In-Progress header should be false</deposit_status_detail>
+ <deposit_external_id>external-id</deposit_external_id>
+</entry>
diff --git a/swh/deposit/tests/data/https_deposit.test.status/1_servicedocument b/swh/deposit/tests/data/https_deposit.test.status/1_servicedocument
new file mode 100644
index 00000000..3abadf1a
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.test.status/1_servicedocument
@@ -0,0 +1,26 @@
+<?xml version="1.0" ?>
+<service xmlns:dcterms="http://purl.org/dc/terms/"
+ xmlns:sword="http://purl.org/net/sword/terms/"
+ xmlns:atom="http://www.w3.org/2005/Atom"
+ xmlns="http://www.w3.org/2007/app">
+
+ <sword:version>2.0</sword:version>
+ <sword:maxUploadSize>209715200</sword:maxUploadSize>
+
+ <workspace>
+ <atom:title>The Software Heritage (SWH) Archive</atom:title>
+ <collection href="test">
+ <atom:title>test Software Collection</atom:title>
+ <accept>application/zip</accept>
+ <accept>application/x-tar</accept>
+ <sword:collectionPolicy>Collection Policy</sword:collectionPolicy>
+ <dcterms:abstract>Software Heritage Archive</dcterms:abstract>
+ <sword:treatment>Collect, Preserve, Share</sword:treatment>
+ <sword:mediation>false</sword:mediation>
+ <sword:metadataRelevantHeader>false</sword:metadataRelevantHeader>
+ <sword:acceptPackaging>http://purl.org/net/sword/package/SimpleZip</sword:acceptPackaging>
+ <sword:service>https://deposit.test.status/1/test/</sword:service>
+ <sword:name>test</sword:name>
+ </collection>
+ </workspace>
+</service>
diff --git a/swh/deposit/tests/data/https_deposit.test.status/1_test_1033_status b/swh/deposit/tests/data/https_deposit.test.status/1_test_1033_status
new file mode 100644
index 00000000..afbc0f4d
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.test.status/1_test_1033_status
@@ -0,0 +1,10 @@
+<entry xmlns="http://www.w3.org/2005/Atom"
+ xmlns:sword="http://purl.org/net/sword/"
+ xmlns:dcterms="http://purl.org/dc/terms/">
+ <deposit_id>1033</deposit_id>
+ <deposit_status>done</deposit_status>
+ <deposit_status_detail>The deposit has been successfully loaded into the Software Heritage archive</deposit_status_detail>
+ <deposit_swh_id>swh:1:dir:ef04a768181417fbc5eef4243e2507915f24deea</deposit_swh_id>
+ <deposit_swh_id_context>swh:1:dir:ef04a768181417fbc5eef4243e2507915f24deea;origin=https://www.softwareheritage.org/check-deposit-2020-10-08T13:52:34.509655;visit=swh:1:snp:c477c6ef51833127b13a86ece7d75e5b3cc4e93d;anchor=swh:1:rev:f26f3960c175f15f6e24200171d446b86f6f7230;path=/</deposit_swh_id_context>
+ <deposit_external_id>check-deposit-2020-10-08T13:52:34.509655</deposit_external_id>
+</entry>
diff --git a/swh/deposit/tests/data/https_deposit.test.updateswhid/1_servicedocument b/swh/deposit/tests/data/https_deposit.test.updateswhid/1_servicedocument
new file mode 100644
index 00000000..81147fa1
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.test.updateswhid/1_servicedocument
@@ -0,0 +1,26 @@
+<?xml version="1.0" ?>
+<service xmlns:dcterms="http://purl.org/dc/terms/"
+ xmlns:sword="http://purl.org/net/sword/terms/"
+ xmlns:atom="http://www.w3.org/2005/Atom"
+ xmlns="http://www.w3.org/2007/app">
+
+ <sword:version>2.0</sword:version>
+ <sword:maxUploadSize>209715200</sword:maxUploadSize>
+
+ <workspace>
+ <atom:title>The Software Heritage (SWH) Archive</atom:title>
+ <collection href="test">
+ <atom:title>test Software Collection</atom:title>
+ <accept>application/zip</accept>
+ <accept>application/x-tar</accept>
+ <sword:collectionPolicy>Collection Policy</sword:collectionPolicy>
+ <dcterms:abstract>Software Heritage Archive</dcterms:abstract>
+ <sword:treatment>Collect, Preserve, Share</sword:treatment>
+ <sword:mediation>false</sword:mediation>
+ <sword:metadataRelevantHeader>false</sword:metadataRelevantHeader>
+ <sword:acceptPackaging>http://purl.org/net/sword/package/SimpleZip</sword:acceptPackaging>
+ <sword:service>https://deposit.test.updateswhid/1/test/</sword:service>
+ <sword:name>test</sword:name>
+ </collection>
+ </workspace>
+</service>
diff --git a/swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_123_metadata b/swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_123_metadata
new file mode 100644
index 00000000..270d91f4
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_123_metadata
@@ -0,0 +1,10 @@
+<entry xmlns="http://www.w3.org/2005/Atom"
+ xmlns:sword="http://purl.org/net/sword/"
+ xmlns:dcterms="http://purl.org/dc/terms/">
+ <deposit_id>123</deposit_id>
+ <deposit_status>done</deposit_status>
+ <deposit_status_detail>The deposit has been successfully loaded into the Software Heritage archive</deposit_status_detail>
+ <deposit_swh_id>swh:1:dir:ef04a768181417fbc5eef4243e2507915f24deea</deposit_swh_id>
+ <deposit_swh_id_context>swh:1:dir:ef04a768181417fbc5eef4243e2507915f24deea;origin=https://www.softwareheritage.org/check-deposit-2020-10-08T13:52:34.509655;visit=swh:1:snp:c477c6ef51833127b13a86ece7d75e5b3cc4e93d;anchor=swh:1:rev:f26f3960c175f15f6e24200171d446b86f6f7230;path=/</deposit_swh_id_context>
+ <deposit_external_id>check-deposit-2020-10-08T13:52:34.509655</deposit_external_id>
+</entry>
diff --git a/swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_123_status b/swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_123_status
new file mode 100644
index 00000000..270d91f4
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_123_status
@@ -0,0 +1,10 @@
+<entry xmlns="http://www.w3.org/2005/Atom"
+ xmlns:sword="http://purl.org/net/sword/"
+ xmlns:dcterms="http://purl.org/dc/terms/">
+ <deposit_id>123</deposit_id>
+ <deposit_status>done</deposit_status>
+ <deposit_status_detail>The deposit has been successfully loaded into the Software Heritage archive</deposit_status_detail>
+ <deposit_swh_id>swh:1:dir:ef04a768181417fbc5eef4243e2507915f24deea</deposit_swh_id>
+ <deposit_swh_id_context>swh:1:dir:ef04a768181417fbc5eef4243e2507915f24deea;origin=https://www.softwareheritage.org/check-deposit-2020-10-08T13:52:34.509655;visit=swh:1:snp:c477c6ef51833127b13a86ece7d75e5b3cc4e93d;anchor=swh:1:rev:f26f3960c175f15f6e24200171d446b86f6f7230;path=/</deposit_swh_id_context>
+ <deposit_external_id>check-deposit-2020-10-08T13:52:34.509655</deposit_external_id>
+</entry>
diff --git a/swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_321_status b/swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_321_status
new file mode 100644
index 00000000..557e2167
--- /dev/null
+++ b/swh/deposit/tests/data/https_deposit.test.updateswhid/1_test_321_status
@@ -0,0 +1,8 @@
+<entry xmlns="http://www.w3.org/2005/Atom"
+ xmlns:sword="http://purl.org/net/sword/"
+ xmlns:dcterms="http://purl.org/dc/terms/">
+ <deposit_id>321</deposit_id>
+ <deposit_status>partial</deposit_status>
+ <deposit_status_detail>The deposit is in partial state</deposit_status_detail>
+ <deposit_external_id>check-deposit-2020-10-08T13:52:34.509655</deposit_external_id>
+</entry>
diff --git a/swh/deposit/tests/loader/conftest.py b/swh/deposit/tests/loader/conftest.py
index 260bd327..2f8f9e6b 100644
--- a/swh/deposit/tests/loader/conftest.py
+++ b/swh/deposit/tests/loader/conftest.py
@@ -1,37 +1,23 @@
# Copyright (C) 2019-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
-from functools import partial
-import re
-
import pytest
-from swh.core.pytest_plugin import get_response_cb
from swh.deposit.loader.checker import DepositChecker
@pytest.fixture
def deposit_config(tmp_path):
return {
"deposit": {
"url": "https://deposit.softwareheritage.org/1/private/",
"auth": {},
}
}
@pytest.fixture
def deposit_checker(deposit_config_path):
return DepositChecker()
-
-
-@pytest.fixture
-def requests_mock_datadir(datadir, requests_mock_datadir):
- """Override default behavior to deal with put method
-
- """
- cb = partial(get_response_cb, datadir=datadir)
- requests_mock_datadir.put(re.compile("https://"), body=cb)
- return requests_mock_datadir
diff --git a/swh/deposit/tests/loader/test_client.py b/swh/deposit/tests/loader/test_client.py
index 55edd2c7..02f4b495 100644
--- a/swh/deposit/tests/loader/test_client.py
+++ b/swh/deposit/tests/loader/test_client.py
@@ -1,246 +1,262 @@
-# Copyright (C) 2017-2019 The Software Heritage developers
+# Copyright (C) 2017-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import json
import os
from typing import Any, Callable, Optional
import unittest
from urllib.parse import urlparse
import pytest
from swh.deposit.client import PrivateApiDepositClient
from swh.deposit.config import DEPOSIT_STATUS_LOAD_FAILURE, DEPOSIT_STATUS_LOAD_SUCCESS
+
CLIENT_TEST_CONFIG = {
"url": "https://nowhere.org/",
"auth": {}, # no authentication in test scenario
}
+@pytest.fixture
+def deposit_config():
+ return CLIENT_TEST_CONFIG
+
+
+def test_client_config(deposit_config_path):
+ for client in [
+ # config passed as constructor parameter
+ PrivateApiDepositClient(config=CLIENT_TEST_CONFIG),
+ # config loaded from environment
+ PrivateApiDepositClient()
+ ]:
+ assert client.config == CLIENT_TEST_CONFIG
+
+
def build_expected_path(datadir, base_url: str, api_url: str) -> str:
"""Build expected path from api to served file
"""
url = urlparse(base_url)
dirname = "%s_%s" % (url.scheme, url.hostname)
if api_url.endswith("/"):
api_url = api_url[:-1]
if api_url.startswith("/"):
api_url = api_url[1:]
suffix_path = api_url.replace("/", "_")
return os.path.join(datadir, dirname, suffix_path)
def test_build_expected_path(datadir):
actual_path = build_expected_path(datadir, "http://example.org", "/hello/you/")
assert actual_path == os.path.join(datadir, "http_example.org", "hello_you")
def read_served_path(
datadir,
base_url: str,
api_url: str,
convert_fn: Optional[Callable[[str], Any]] = None,
) -> bytes:
"""Read served path
"""
archive_path = build_expected_path(datadir, base_url, api_url)
with open(archive_path, "rb") as f:
content = f.read()
if convert_fn:
content = convert_fn(content.decode("utf-8"))
return content
def test_read_served_path(datadir):
actual_content = read_served_path(datadir, "http://example.org", "/hello/you/")
assert actual_content == b"hello people\n"
actual_content2 = read_served_path(
datadir, "http://example.org", "/hello.json", convert_fn=json.loads
)
assert actual_content2 == {"a": [1, 3]}
# private api to retrieve archive
def test_archive_get(tmp_path, datadir, requests_mock_datadir):
"""Retrieving archive data through private api should stream data
"""
api_url = "/1/private/test/1/raw/"
- client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
+ client = PrivateApiDepositClient(CLIENT_TEST_CONFIG)
expected_content = read_served_path(datadir, client.base_url, api_url)
archive_path = os.path.join(tmp_path, "test.archive")
archive_path = client.archive_get(api_url, archive_path)
assert os.path.exists(archive_path) is True
with open(archive_path, "rb") as f:
actual_content = f.read()
assert actual_content == expected_content
assert client.base_url == CLIENT_TEST_CONFIG["url"]
assert client.auth is None
def test_archive_get_auth(tmp_path, datadir, requests_mock_datadir):
"""Retrieving archive data through private api should stream data
"""
api_url = "/1/private/test/1/raw/"
config = CLIENT_TEST_CONFIG.copy()
config["auth"] = { # add authentication setup
"username": "user",
"password": "pass",
}
client = PrivateApiDepositClient(config)
expected_content = read_served_path(datadir, client.base_url, api_url)
archive_path = os.path.join(tmp_path, "test.archive")
archive_path = client.archive_get(api_url, archive_path)
assert os.path.exists(archive_path) is True
with open(archive_path, "rb") as f:
actual_content = f.read()
assert actual_content == expected_content
assert client.base_url == CLIENT_TEST_CONFIG["url"]
assert client.auth == ("user", "pass")
def test_archive_get_ko(tmp_path, datadir, requests_mock_datadir):
"""Reading archive can fail for some reasons
"""
unknown_api_url = "/1/private/unknown/deposit-id/raw/"
client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
with pytest.raises(ValueError, match="Problem when retrieving deposit"):
client.archive_get(unknown_api_url, "some/path")
# private api read metadata
def test_metadata_get(datadir, requests_mock_datadir):
"""Reading archive should write data in temporary directory
"""
api_url = "/1/private/test/1/metadata"
client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
actual_metadata = client.metadata_get(api_url)
assert isinstance(actual_metadata, str) is False
expected_content = read_served_path(
datadir, client.base_url, api_url, convert_fn=json.loads
)
assert actual_metadata == expected_content
def test_metadata_get_ko(requests_mock_datadir):
"""Reading metadata can fail for some reasons
"""
unknown_api_url = "/1/private/unknown/deposit-id/metadata/"
client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
with pytest.raises(ValueError, match="Problem when retrieving metadata"):
client.metadata_get(unknown_api_url)
# private api check
def test_check(requests_mock_datadir):
"""When check ok, this should return the deposit's status
"""
api_url = "/1/private/test/1/check"
client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
r = client.check(api_url)
assert r == "something"
def test_check_fails(requests_mock_datadir):
"""Checking deposit can fail for some reason
"""
unknown_api_url = "/1/private/test/10/check"
client = PrivateApiDepositClient(config=CLIENT_TEST_CONFIG)
with pytest.raises(ValueError, match="Problem when checking deposit"):
client.check(unknown_api_url)
# private api update status
class FakeRequestClientPut:
"""Fake Request client dedicated to put request method calls.
"""
args = None
kwargs = None
def put(self, *args, **kwargs):
self.args = args
self.kwargs = kwargs
class PrivateApiDepositClientStatusUpdateTest(unittest.TestCase):
def test_status_update(self):
"""Update status
"""
_client = FakeRequestClientPut()
deposit_client = PrivateApiDepositClient(
config=CLIENT_TEST_CONFIG, _client=_client
)
deposit_client.status_update(
"/update/status",
DEPOSIT_STATUS_LOAD_SUCCESS,
revision_id="some-revision-id",
)
self.assertEqual(_client.args, ("https://nowhere.org/update/status",))
self.assertEqual(
_client.kwargs,
{
"json": {
"status": DEPOSIT_STATUS_LOAD_SUCCESS,
"revision_id": "some-revision-id",
}
},
)
def test_status_update_with_no_revision_id(self):
"""Reading metadata can fail for some reasons
"""
_client = FakeRequestClientPut()
deposit_client = PrivateApiDepositClient(
config=CLIENT_TEST_CONFIG, _client=_client
)
deposit_client.status_update("/update/status/fail", DEPOSIT_STATUS_LOAD_FAILURE)
self.assertEqual(_client.args, ("https://nowhere.org/update/status/fail",))
self.assertEqual(
_client.kwargs, {"json": {"status": DEPOSIT_STATUS_LOAD_FAILURE,}}
)
diff --git a/tox.ini b/tox.ini
index 625647a4..a6efef96 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,45 +1,45 @@
[tox]
envlist=flake8,mypy,py3-django2
[testenv]
extras =
testing
deps =
# the dependency below is needed for now as a workaround for
# https://github.com/pypa/pip/issues/6239
swh.core[http] >= 0.3
swh.scheduler[testing] >= 0.5.0
dev: pdbpp
pytest-cov
django2: Django>=2,<3
commands =
pytest \
!dev: --cov {envsitepackagesdir}/swh/deposit --cov-branch \
{envsitepackagesdir}/swh/deposit \
{posargs}
[testenv:black]
skip_install = true
deps =
- black
+ black==19.10b0
commands =
{envpython} -m black --check swh
[testenv:flake8]
skip_install = true
deps =
flake8
commands =
{envpython} -m flake8 \
--exclude=.tox,.git,__pycache__,.tox,.eggs,*.egg,swh/deposit/migrations
[testenv:mypy]
setenv = DJANGO_SETTINGS_MODULE=swh.deposit.settings.testing
extras =
testing
deps =
mypy
django-stubs
djangorestframework-stubs
commands =
mypy swh
File Metadata
Details
Attached
Mime Type
text/x-diff
Expires
Fri, Jul 4, 5:24 PM (3 w, 3 d ago)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
3452633
Attached To
rDDEP Push deposit
Event Timeline
Log In to Comment