Page MenuHomeSoftware Heritage
Differential D1416

deposit: Adapt revision dates setting according to new rules
ClosedPublic
Actions

Authored by ardumont on Apr 13 2019, 12:25 PM.

Details

Reviewers
moranegg
douardda
Group Reviewers
Reviewers
Commits
rDDEPc183276262e1: Refactor: Extract date utils manipulation function into utils module
rDDEPa7a7170df586: utils: Make parsing date result in a reasonable date
rDDEP00a8abd44f36: tests: Factorize metadata xml duplication to explicit tested date
rDDEP0138edcadef7: deposit_read: Refactor to clarify intents and docstring
rDDEP37b48717a83f: loader: Use metadata date as author/committer date if provided
Summary

Use respectively codemeta.dateCreated/codemeta.datePublished as revision.author_date/revision.committer_date if provided.
Else use codemeta.dateCreated for those field dates if provided
Else use codemeta.datePublished for those field dates if provided
Else use deposit.complete_date for those fields

Related T1637

Test Plan

tox

Diff Detail

Repository
rDDEP Push deposit
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Apr 13 2019, 12:25 PM
swh-public-ci added a comment.Apr 13 2019, 12:28 PM

Build is green
See https://jenkins.softwareheritage.org/job/DDEP/job/tox/98/ for more details.

Harbormaster completed remote builds in B5427: Diff 4565.Apr 13 2019, 12:28 PM
ardumont added inline comments.Apr 13 2019, 12:45 PM
swh/deposit/tests/loader/test_loader.py
79

This was actually wrong to trigger a load of a deposit with a wrong status (partial here which is only possible in tests).

ardumont updated this revision to Diff 4566.Apr 13 2019, 12:45 PM

Remove spurrious print statements

swh-public-ci added a comment.Apr 13 2019, 12:49 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tox/99/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tox/99/console

Harbormaster failed remote builds in B5428: Diff 4566!Apr 13 2019, 12:49 PM
ardumont updated this revision to Diff 4570.Apr 15 2019, 10:36 AM

Fix irrelevant order failure in deposit check tests

swh-public-ci added a comment.Apr 15 2019, 10:39 AM

Build is green
See https://jenkins.softwareheritage.org/job/DDEP/job/tox/100/ for more details.

Harbormaster completed remote builds in B5432: Diff 4570.Apr 15 2019, 10:39 AM
ardumont added inline comments.Apr 15 2019, 10:45 AM
swh/deposit/tests/loader/test_loader.py
79

That's why i changed the test ;)

moranegg added a comment.Apr 15 2019, 10:55 AM

I started reviewing but the question is more complex than I first stated
I asked HAL to fix dateCreated to their real metadata 'date de création/écriture'
and to add datePublished as the deposit date
While dateCreated isn't mandatory on HAL, datePublished is implicitly mandatory
And the date of arrival to SWH isn't the same of datePublished, because of the moderation process on HAL side
It can differ by hours, days or even weeks

swh/deposit/api/private/deposit_check.py
130 ↗(On Diff #4570)

I'm against making this metadata obligatory because I think it will break the chain with HAL

swh/deposit/tests/api/test_deposit_check.py
141 ↗(On Diff #4570)

again, shouldn't be mandatory

ardumont planned changes to this revision.Apr 15 2019, 2:30 PM

The diff is outdated now.
This needs rework.

swh/deposit/api/private/deposit_read.py
217–218

According to our discussion anyway, it's no longer the commit date, it's the author date.

ardumont updated this revision to Diff 4584.Apr 15 2019, 6:21 PM

Use new committer/author dates rules to create the revision's date entries

swh-public-ci added a comment.Apr 15 2019, 6:24 PM

Build is green
See https://jenkins.softwareheritage.org/job/DDEP/job/tox/102/ for more details.

Harbormaster completed remote builds in B5447: Diff 4584.Apr 15 2019, 6:24 PM
ardumont retitled this revision from deposit: Change commit date to the date of software creation (instead of the deposit's completed date) to deposit: Adapt revision dates setting according to new rules.Apr 16 2019, 9:54 AM
ardumont edited the summary of this revision. (Show Details)
ardumont added a project: SWORD deposit.
moranegg added a comment.Apr 16 2019, 12:28 PM

Seems great!
tox passes locally.
I would add/change the date formats in tests (because we can recieve dates in different styles)
and/or test only the _parse_date and compute_date functions.
Here some examples:

<date type="whenSubmitted">2018-02-27 10:45:10</date>
<date type="whenWritten">2012</date>
<date type="whenModified">2018-06-11 17:02:02</date>
<date type="whenReleased">2018-03-20 13:09:08</date>
<date type="whenProduced">2012</date>
<date type="whenEndEmbargoed">2018-02-27</date>

This is directly from HAL's TEI, which shows the different formats it can send.

douardda added inline comments.Apr 16 2019, 12:35 PM
swh/deposit/api/private/deposit_read.py
166–176

nitpicking a bit, but IMHO the t temp var is not needed here. The whole if/elif block can manipulate {author,commit}_date variables directly.

swh/deposit/tests/api/test_deposit_read_metadata.py
399

Couldn't we share this big chunk of xml among tests? I'm a big advocate of copy/paste in tests (rather than clever abstractions/genericity that make the test hard to read for a human), but in this case, it's hard to guess what the differences are between these tests. The docstrings/comments do describe them, but ...

ardumont added inline comments.Apr 16 2019, 1:49 PM
swh/deposit/api/private/deposit_read.py
166–176

oh, righty right.
initially, i just returned the tuple t and did not do anything on it
up until i realized i needed to do stuff on it...

swh/deposit/tests/api/test_deposit_read_metadata.py
399

I'll see what i can do, it sure irritates me a lot...
I was focused on making the thing works first ;)

ardumont added a comment.Apr 16 2019, 1:53 PM

I would add/change the date formats in tests (because we can recieve dates in different styles)
and/or test only the _parse_date and compute_date functions.
Here some examples:

<date type="whenSubmitted">2018-02-27 10:45:10</date>
<date type="whenWritten">2012</date>
<date type="whenModified">2018-06-11 17:02:02</date>
<date type="whenReleased">2018-03-20 13:09:08</date>
<date type="whenProduced">2012</date>
<date type="whenEndEmbargoed">2018-02-27</date>

I'm pretty sure some of those formats are not parsed correctly already.
The ones with only the year for example gives:

In [1]: from dateutil.parser import parse

In [2]: date = '2012'

In [3]: parse(date)
Out[3]: datetime.datetime(2012, 4, 16, 0, 0)

The others are fine though.

ardumont updated this revision to Diff 4596.Apr 16 2019, 2:39 PM
  • deposit_read: Refactor to clarify intents and docstring

Also rebase on latest master

swh-public-ci added a comment.Apr 16 2019, 2:42 PM

Build is green
See https://jenkins.softwareheritage.org/job/DDEP/job/tox/107/ for more details.

Harbormaster completed remote builds in B5461: Diff 4596.Apr 16 2019, 2:42 PM
ardumont updated this revision to Diff 4597.Apr 16 2019, 2:50 PM
  • tests: Factorize metadata xml duplication to explicit tested date
swh-public-ci added a comment.Apr 16 2019, 2:53 PM

Build is green
See https://jenkins.softwareheritage.org/job/DDEP/job/tox/108/ for more details.

Harbormaster completed remote builds in B5462: Diff 4597.Apr 16 2019, 2:53 PM
ardumont updated this revision to Diff 4598.Apr 16 2019, 3:11 PM
  • Refactor: Extract date utils manipulation function into utils module
ardumont added inline comments.Apr 16 2019, 3:13 PM
swh/deposit/tests/test_utils.py
182 ↗(On Diff #4598)

here!

swh-public-ci added a comment.Apr 16 2019, 3:15 PM

Build is green
See https://jenkins.softwareheritage.org/job/DDEP/job/tox/109/ for more details.

Harbormaster completed remote builds in B5463: Diff 4598.Apr 16 2019, 3:15 PM
moranegg added a comment.Apr 16 2019, 3:21 PM
======================= 112 passed, 1 warnings in 16.33 seconds ========================
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7ff254794840>
Traceback (most recent call last):
  File "/home/morane/Documents/code/swh-environment/swh-deposit/.tox/py3/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7ff254794840>
Traceback (most recent call last):
  File "/home/morane/Documents/code/swh-environment/swh-deposit/.tox/py3/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
WARNING [pifpaf.drivers] `psutil.Popen(pid=5224, status='terminated')` is already gone, sending SIGKILL to its process group
WARNING [pifpaf.drivers] `psutil.Popen(pid=5207, status='terminated')` is already gone, sending SIGKILL to its process group
WARNING [pifpaf.drivers] `psutil.Popen(pid=5202, status='terminated')` is already gone, sending SIGKILL to its process group
_______________________________________ summary ________________________________________
  flake8: commands succeeded
  py3: commands succeeded
  congratulations :)

Looks good, waiting for the 182 line comment about the choice of a date when only a year is entered .

swh/deposit/tests/api/test_deposit_read_metadata.py
473

The design choice of taking the first date in the list is good for me

swh/deposit/tests/test_utils.py
146 ↗(On Diff #4598)

that's good.

163 ↗(On Diff #4598)

tha's good, no?

180 ↗(On Diff #4598)

should be 2017-01-01 00:00:00, if possible?

ardumont updated this revision to Diff 4600.Apr 16 2019, 5:02 PM
  • utils: Make parsing date result in a reasonable date
swh-public-ci added a comment.Apr 16 2019, 5:05 PM

Build is green
See https://jenkins.softwareheritage.org/job/DDEP/job/tox/110/ for more details.

Harbormaster completed remote builds in B5465: Diff 4600.Apr 16 2019, 5:05 PM
ardumont updated this revision to Diff 4604.Apr 16 2019, 7:26 PM

Update test docstring to match the reality of the test

swh-public-ci added a comment.Apr 16 2019, 7:29 PM

Build is green
See https://jenkins.softwareheritage.org/job/DDEP/job/tox/112/ for more details.

Harbormaster completed remote builds in B5470: Diff 4604.Apr 16 2019, 7:29 PM
ardumont added inline comments.Apr 17 2019, 9:34 AM
swh/deposit/tests/test_utils.py
180 ↗(On Diff #4598)

That's fixed now thanks to the discussion with @douardda (iso8601)

ardumont mentioned this in T1638: Deposit: error when submitting through cli.Apr 17 2019, 10:02 AM
ardumont updated this revision to Diff 4606.Apr 17 2019, 10:46 AM

Plug on master branch

swh-public-ci added a comment.Apr 17 2019, 10:51 AM

Build is green
See https://jenkins.softwareheritage.org/job/DDEP/job/tox/114/ for more details.

Harbormaster completed remote builds in B5472: Diff 4606.Apr 17 2019, 10:51 AM
This revision was not accepted when it landed; it landed in state Needs Review.Apr 17 2019, 10:52 AM
Closed by commit rDDEP37b48717a83f: loader: Use metadata date as author/committer date if provided (authored by ardumont). · Explain Why
This revision was automatically updated to reflect the committed changes.
ardumont added a comment.Apr 17 2019, 11:17 AM

This revision was not accepted when it landed; it landed in state Needs Review.Wed, Apr 17, 10:52

seen with @douardda orally ;)

Revision Contents
Changeset List

PathSize
swh/
deposit/
api/
private/
49 lines
tests/
api/
516 lines
1 line
loader/
20 lines

Diff 4607

View Options

swh/deposit/api/private/deposit_read.py

# Copyright (C) 2017-2018 The Software Heritage developers # Copyright (C) 2017-2019 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import json import json
import os import os
import shutil import shutil
import tempfile import tempfile
from contextlib import contextmanager from contextlib import contextmanager
from dateutil import parser
from django.http import FileResponse from django.http import FileResponse
from rest_framework import status from rest_framework import status
from swh.core import tarball from swh.core import tarball
from swh.model import identifiers from swh.model import identifiers
from . import DepositReadMixin from . import DepositReadMixin
from ...config import SWH_PERSON, ARCHIVE_TYPE from ...config import SWH_PERSON, ARCHIVE_TYPE
▲ Show 20 Lines • Show All 111 Lines • ▼ Show 20 Lines """
def _retrieve_url(self, deposit, metadata): def _retrieve_url(self, deposit, metadata):
client_domain = deposit.client.domain client_domain = deposit.client.domain
for field in metadata: for field in metadata:
if 'url' in field: if 'url' in field:
if client_domain in metadata[field]: if client_domain in metadata[field]:
return metadata[field] return metadata[field]
def _prepare_date(self, date):
"""Prepare date fields as normalized swh date
"""
if isinstance(date, list):
date = date[0]
if isinstance(date, str):
date = parser.parse(date)
return identifiers.normalize_timestamp(date)
def _compute_date(self, deposit, metadata):
"""Compute the date to use as a tuple of author date, committer date.
Each of those date are swh normalized immediately.
Args:
deposit (Deposit): Deposit model representation
metadata (Dict): Metadata dict representation
Returns:
Tuple of author date, committer date. Those dates are
swh normalized.
"""
commit_date = metadata.get('codemeta:datePublished')
author_date = metadata.get('codemeta:dateCreated')
if author_date and commit_date:
t = (author_date, commit_date)
elif commit_date:
t = (commit_date, commit_date)
elif author_date:
t = (author_date, author_date)
else:
date = deposit.complete_date
t = (date, date)
return (
douarddaUnsubmitted
Not Done
Inline Actions

nitpicking a bit, but IMHO the t temp var is not needed here. The whole if/elif block can manipulate {author,commit}_date variables directly.

douardda: nitpicking a bit, but IMHO the `t` temp var is not needed here. The whole if/elif block can…
ardumontAuthorUnsubmitted
Done
Inline Actions

oh, righty right.
initially, i just returned the tuple t and did not do anything on it
up until i realized i needed to do stuff on it...

ardumont: oh, righty right. initially, i just returned the tuple `t` and did not do anything on it up…
self._prepare_date(t[0]), self._prepare_date(t[1]))
def metadata_read(self, deposit): def metadata_read(self, deposit):
"""Read and aggregate multiple data on deposit into one unified data """Read and aggregate multiple data on deposit into one unified data
dictionary. dictionary.
Args: Args:
deposit (Deposit): Deposit concerned by the data aggregation. deposit (Deposit): Deposit concerned by the data aggregation.
Returns: Returns:
Show All 17 Lines def metadata_read(self, deposit):
# metadata provider # metadata provider
self.provider['provider_name'] = deposit.client.last_name self.provider['provider_name'] = deposit.client.last_name
self.provider['provider_url'] = deposit.client.provider_url self.provider['provider_url'] = deposit.client.provider_url
revision_type = 'tar' revision_type = 'tar'
revision_msg = '%s: Deposit %s in collection %s' % ( revision_msg = '%s: Deposit %s in collection %s' % (
fullname, deposit.id, deposit.collection.name) fullname, deposit.id, deposit.collection.name)
complete_date = identifiers.normalize_timestamp(deposit.complete_date)
author_date, commit_date = self._compute_date(deposit, metadata)
data['revision'] = { data['revision'] = {
'synthetic': True, 'synthetic': True,
'date': complete_date, 'date': author_date,
'committer_date': complete_date, 'committer_date': commit_date,
ardumontAuthorUnsubmitted
Done
Inline Actions

According to our discussion anyway, it's no longer the commit date, it's the author date.

ardumont: According to our discussion anyway, it's no longer the commit date, it's the author date.
'author': author_committer, 'author': author_committer,
'committer': author_committer, 'committer': author_committer,
'type': revision_type, 'type': revision_type,
'message': revision_msg, 'message': revision_msg,
'metadata': metadata, 'metadata': metadata,
} }
if deposit.parent: if deposit.parent:
Show All 24 Lines
View Options

swh/deposit/tests/api/test_deposit_read_metadata.py

Show First 20 Lines • Show All 44 Lines • ▼ Show 20 Lines def test_read_metadata(self):
'url': 'https://hal-test.archives-ouvertes.fr/' + 'url': 'https://hal-test.archives-ouvertes.fr/' +
'some-external-id', 'some-external-id',
'type': 'deposit' 'type': 'deposit'
}, },
'origin_metadata': { 'origin_metadata': {
'metadata': { 'metadata': {
'@xmlns': ['http://www.w3.org/2005/Atom'], '@xmlns': ['http://www.w3.org/2005/Atom'],
'author': ['some awesome author', 'another one', 'no one'], 'author': ['some awesome author', 'another one', 'no one'],
'codemeta:dateCreated': '2017-10-07T15:17:08Z',
'external_identifier': 'some-external-id', 'external_identifier': 'some-external-id',
'url': 'https://hal-test.archives-ouvertes.fr/' + 'url': 'https://hal-test.archives-ouvertes.fr/' +
'some-external-id' 'some-external-id'
}, },
'provider': { 'provider': {
'provider_name': 'hal', 'provider_name': 'hal',
'provider_type': 'deposit_client', 'provider_type': 'deposit_client',
'provider_url': 'https://hal-test.archives-ouvertes.fr/', 'provider_url': 'https://hal-test.archives-ouvertes.fr/',
'metadata': {} 'metadata': {}
}, },
'tool': { 'tool': {
'name': 'swh-deposit', 'name': 'swh-deposit',
'version': '0.0.1', 'version': '0.0.1',
'configuration': { 'configuration': {
'sword_version': '2' 'sword_version': '2'
} }
} }
}, },
'revision': { 'revision': {
'synthetic': True, 'synthetic': True,
'committer_date': None, 'committer_date': {
'timestamp': {
'seconds': 1507389428,
'microseconds': 0
},
'offset': 0,
'negative_utc': False
},
'message': 'hal: Deposit %s in collection hal' % deposit_id, 'message': 'hal: Deposit %s in collection hal' % deposit_id,
'author': SWH_PERSON, 'author': SWH_PERSON,
'committer': SWH_PERSON, 'committer': SWH_PERSON,
'date': None, 'date': {
'timestamp': {
'seconds': 1507389428,
'microseconds': 0
},
'offset': 0,
'negative_utc': False
},
'metadata': { 'metadata': {
'@xmlns': ['http://www.w3.org/2005/Atom'], '@xmlns': ['http://www.w3.org/2005/Atom'],
'author': ['some awesome author', 'another one', 'no one'], 'author': ['some awesome author', 'another one', 'no one'],
'external_identifier': 'some-external-id', 'external_identifier': 'some-external-id',
'codemeta:dateCreated': '2017-10-07T15:17:08Z',
'url': 'https://hal-test.archives-ouvertes.fr/' + 'url': 'https://hal-test.archives-ouvertes.fr/' +
'some-external-id' 'some-external-id'
}, },
'type': 'tar' 'type': 'tar'
}, },
'branch_name': 'master', 'branch_name': 'master',
} }
Show All 40 Lines def test_read_metadata_revision_with_parent(self):
'url': 'https://hal-test.archives-ouvertes.fr/' + 'url': 'https://hal-test.archives-ouvertes.fr/' +
'some-external-id', 'some-external-id',
'type': 'deposit' 'type': 'deposit'
}, },
'origin_metadata': { 'origin_metadata': {
'metadata': { 'metadata': {
'@xmlns': ['http://www.w3.org/2005/Atom'], '@xmlns': ['http://www.w3.org/2005/Atom'],
'author': ['some awesome author', 'another one', 'no one'], 'author': ['some awesome author', 'another one', 'no one'],
'codemeta:dateCreated': '2017-10-07T15:17:08Z',
'external_identifier': 'some-external-id', 'external_identifier': 'some-external-id',
'url': 'https://hal-test.archives-ouvertes.fr/' + 'url': 'https://hal-test.archives-ouvertes.fr/' +
'some-external-id' 'some-external-id'
}, },
'provider': { 'provider': {
'provider_name': 'hal', 'provider_name': 'hal',
'provider_type': 'deposit_client', 'provider_type': 'deposit_client',
'provider_url': 'https://hal-test.archives-ouvertes.fr/', 'provider_url': 'https://hal-test.archives-ouvertes.fr/',
'metadata': {} 'metadata': {}
}, },
'tool': { 'tool': {
'name': 'swh-deposit', 'name': 'swh-deposit',
'version': '0.0.1', 'version': '0.0.1',
'configuration': { 'configuration': {
'sword_version': '2' 'sword_version': '2'
} }
} }
}, },
'revision': { 'revision': {
'synthetic': True, 'synthetic': True,
'date': None, 'date': {
'committer_date': None, 'timestamp': {
'seconds': 1507389428,
'microseconds': 0
},
'offset': 0,
'negative_utc': False
},
'committer_date': {
'timestamp': {
'seconds': 1507389428,
'microseconds': 0
},
'offset': 0,
'negative_utc': False
},
'author': SWH_PERSON, 'author': SWH_PERSON,
'committer': SWH_PERSON, 'committer': SWH_PERSON,
'type': 'tar', 'type': 'tar',
'message': 'hal: Deposit %s in collection hal' % deposit_id, 'message': 'hal: Deposit %s in collection hal' % deposit_id,
'metadata': { 'metadata': {
'@xmlns': ['http://www.w3.org/2005/Atom'], '@xmlns': ['http://www.w3.org/2005/Atom'],
'author': ['some awesome author', 'another one', 'no one'], 'author': ['some awesome author', 'another one', 'no one'],
'codemeta:dateCreated': '2017-10-07T15:17:08Z',
'external_identifier': 'some-external-id', 'external_identifier': 'some-external-id',
'url': 'https://hal-test.archives-ouvertes.fr/' + 'url': 'https://hal-test.archives-ouvertes.fr/' +
'some-external-id' 'some-external-id'
}, },
'parents': [swh_id] 'parents': [swh_id]
}, },
'branch_name': 'master', 'branch_name': 'master',
} }
self.assertEqual(data, expected_meta) self.assertEqual(data, expected_meta)
def test_read_metadata_3(self):
"""dateCreated/datePublished provided, revision uses author/committer date
"""
# add metadata to the deposit with datePublished and dateCreated
codemeta_entry_data = b"""<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom"
xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0">
<title>Composing a Web of Audio Applications</title>
<client>hal</client>
<id>hal-01243065</id>
<external_identifier>hal-01243065</external_identifier>
<codemeta:url>https://hal-test.archives-ouvertes.fr/hal-01243065</codemeta:url>
<codemeta:applicationCategory>test</codemeta:applicationCategory>
<codemeta:keywords>DSP programming,Web</codemeta:keywords>
<codemeta:dateCreated>2015-04-06T17:08:47+02:00</codemeta:dateCreated>
<codemeta:description>this is the description</codemeta:description>
<codemeta:version>1</codemeta:version>
<codemeta:runtimePlatform>phpstorm</codemeta:runtimePlatform>
<codemeta:developmentStatus>stable</codemeta:developmentStatus>
<codemeta:programmingLanguage>php</codemeta:programmingLanguage>
<codemeta:programmingLanguage>python</codemeta:programmingLanguage>
<codemeta:programmingLanguage>C</codemeta:programmingLanguage>
<codemeta:datePublished>2017-05-03T16:08:47+02:00</codemeta:datePublished>
<codemeta:license>
<codemeta:name>GNU General Public License v3.0 only</codemeta:name>
</codemeta:license>
<codemeta:license>
<codemeta:name>CeCILL Free Software License Agreement v1.1</codemeta:name>
</codemeta:license>
<author>
<name>HAL</name>
<email>hal@ccsd.cnrs.fr</email>
</author>
<codemeta:author>
<codemeta:name>Morane Gruenpeter</codemeta:name>
</codemeta:author>
</entry>""" # noqa
deposit_id = self.create_deposit_partial_with_data_in_args(
codemeta_entry_data)
url = reverse(PRIVATE_GET_DEPOSIT_METADATA,
args=[self.collection.name, deposit_id])
response = self.client.get(url)
self.assertEqual(response.status_code,
status.HTTP_200_OK)
self.assertEqual(response._headers['content-type'][1],
'application/json')
data = response.json()
expected_origin = {
'type': 'deposit',
'url': 'https://hal-test.archives-ouvertes.fr/hal-01243065'
}
expected_metadata = {
'@xmlns': 'http://www.w3.org/2005/Atom',
'@xmlns:codemeta':
'https://doi.org/10.5063/SCHEMA/CODEMETA-2.0',
'author': {
'email': 'hal@ccsd.cnrs.fr',
'name': 'HAL'
},
'client': 'hal',
'codemeta:applicationCategory': 'test',
'codemeta:author': {
'codemeta:name': 'Morane Gruenpeter'
},
'codemeta:dateCreated': '2015-04-06T17:08:47+02:00',
'codemeta:datePublished': '2017-05-03T16:08:47+02:00',
'codemeta:description': 'this is the description',
'codemeta:developmentStatus': 'stable',
'codemeta:keywords': 'DSP programming,Web',
'codemeta:license': [
{
'codemeta:name': 'GNU General Public License v3.0 only'
},
{
'codemeta:name':
'CeCILL Free Software License Agreement v1.1'
}
],
'codemeta:programmingLanguage': [
'php', 'python', 'C'
],
'codemeta:runtimePlatform': 'phpstorm',
'codemeta:url': 'https://hal-test.archives-ouvertes.fr/hal-01243065', # noqa
'codemeta:version': '1',
'external_identifier': 'hal-01243065',
'id': 'hal-01243065',
'title': 'Composing a Web of Audio Applications'
}
expected_origin_metadata = {
'metadata': expected_metadata,
'provider': {
'metadata': {},
'provider_name': 'hal',
'provider_type': 'deposit_client',
'provider_url': 'https://hal-test.archives-ouvertes.fr/'
},
'tool': {
'configuration': {
'sword_version': '2'
},
'name': 'swh-deposit',
'version': '0.0.1'
}
}
expected_revision = {
'author': {
'email': 'robot@softwareheritage.org',
'fullname': 'Software Heritage',
'name': 'Software Heritage'
},
'committer': {
'email': 'robot@softwareheritage.org',
'fullname': 'Software Heritage',
'name': 'Software Heritage'
},
'committer_date': {
'negative_utc': False,
'offset': 120,
'timestamp': {
'microseconds': 0,
'seconds': 1493820527
}
},
'date': {
'negative_utc': False,
'offset': 120,
'timestamp': {
'microseconds': 0,
'seconds': 1428332927
}
},
'message': 'hal: Deposit %s in collection hal' % deposit_id,
'metadata': expected_metadata,
'synthetic': True,
'type': 'tar'
}
expected_meta = {
'branch_name': 'master',
'origin': expected_origin,
'origin_metadata': expected_origin_metadata,
'revision': expected_revision,
}
self.assertEqual(data, expected_meta)
def test_read_metadata_4(self):
"""dateCreated/datePublished not provided, revision uses complete_date
"""
# add metadata to the deposit with datePublished and dateCreated
codemeta_entry_data = b"""<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom"
xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0">
<title>Composing a Web of Audio Applications</title>
<client>hal</client>
<id>hal-01243065</id>
<external_identifier>hal-01243065</external_identifier>
<codemeta:url>https://hal-test.archives-ouvertes.fr/hal-01243065</codemeta:url>
<codemeta:applicationCategory>test</codemeta:applicationCategory>
<codemeta:keywords>DSP programming</codemeta:keywords>
<codemeta:description>this is the description</codemeta:description>
<codemeta:version>1</codemeta:version>
<codemeta:runtimePlatform>phpstorm</codemeta:runtimePlatform>
<codemeta:developmentStatus>stable</codemeta:developmentStatus>
<codemeta:programmingLanguage>php</codemeta:programmingLanguage>
<codemeta:programmingLanguage>python</codemeta:programmingLanguage>
<codemeta:programmingLanguage>C</codemeta:programmingLanguage>
<codemeta:license>
<codemeta:name>GNU General Public License v3.0 only</codemeta:name>
</codemeta:license>
<codemeta:license>
<codemeta:name>CeCILL Free Software License Agreement v1.1</codemeta:name>
</codemeta:license>
<author>
<name>HAL</name>
<email>hal@ccsd.cnrs.fr</email>
</author>
<codemeta:author>
<codemeta:name>Morane Gruenpeter</codemeta:name>
</codemeta:author>
</entry>""" # noqa
douarddaUnsubmitted
Not Done
Inline Actions

Couldn't we share this big chunk of xml among tests? I'm a big advocate of copy/paste in tests (rather than clever abstractions/genericity that make the test hard to read for a human), but in this case, it's hard to guess what the differences are between these tests. The docstrings/comments do describe them, but ...

douardda: Couldn't we share this big chunk of xml among tests? I'm a big advocate of copy/paste in tests…
ardumontAuthorUnsubmitted
Done
Inline Actions

I'll see what i can do, it sure irritates me a lot...
I was focused on making the thing works first ;)

ardumont: I'll see what i can do, it sure irritates me a lot... I was focused on making the thing works…
deposit_id = self.create_deposit_partial_with_data_in_args(
codemeta_entry_data)
# will use the deposit completed date as fallback date
deposit = Deposit.objects.get(pk=deposit_id)
deposit.complete_date = '2016-04-06'
deposit.save()
url = reverse(PRIVATE_GET_DEPOSIT_METADATA,
args=[self.collection.name, deposit_id])
response = self.client.get(url)
self.assertEqual(response.status_code,
status.HTTP_200_OK)
self.assertEqual(response._headers['content-type'][1],
'application/json')
data = response.json()
expected_origin = {
'type': 'deposit',
'url': 'https://hal-test.archives-ouvertes.fr/hal-01243065'
}
expected_metadata = {
'@xmlns': 'http://www.w3.org/2005/Atom',
'@xmlns:codemeta':
'https://doi.org/10.5063/SCHEMA/CODEMETA-2.0',
'author': {
'email': 'hal@ccsd.cnrs.fr',
'name': 'HAL'
},
'client': 'hal',
'codemeta:applicationCategory': 'test',
'codemeta:author': {
'codemeta:name': 'Morane Gruenpeter'
},
'codemeta:description': 'this is the description',
'codemeta:developmentStatus': 'stable',
'codemeta:keywords': 'DSP programming',
'codemeta:license': [
{
'codemeta:name': 'GNU General Public License v3.0 only'
},
{
'codemeta:name':
'CeCILL Free Software License Agreement v1.1'
}
],
'codemeta:programmingLanguage': [
'php', 'python', 'C'
],
'codemeta:runtimePlatform': 'phpstorm',
'codemeta:url': 'https://hal-test.archives-ouvertes.fr/hal-01243065', # noqa
'codemeta:version': '1',
'external_identifier': 'hal-01243065',
'id': 'hal-01243065',
'title': 'Composing a Web of Audio Applications'
}
expected_origin_metadata = {
'metadata': expected_metadata,
'provider': {
'metadata': {},
'provider_name': 'hal',
'provider_type': 'deposit_client',
'provider_url': 'https://hal-test.archives-ouvertes.fr/'
},
'tool': {
'configuration': {
'sword_version': '2'
},
'name': 'swh-deposit',
'version': '0.0.1'
moraneggUnsubmitted
Not Done
Inline Actions

The design choice of taking the first date in the list is good for me

moranegg: The design choice of taking the first date in the list is good for me
}
}
expected_revision = {
'author': {
'email': 'robot@softwareheritage.org',
'fullname': 'Software Heritage',
'name': 'Software Heritage'
},
'committer': {
'email': 'robot@softwareheritage.org',
'fullname': 'Software Heritage',
'name': 'Software Heritage'
},
'committer_date': {
'negative_utc': False,
'offset': 0,
'timestamp': {
'microseconds': 0,
'seconds': 1459900800
}
},
'date': {
'negative_utc': False,
'offset': 0,
'timestamp': {
'microseconds': 0,
'seconds': 1459900800
}
},
'message': 'hal: Deposit %s in collection hal' % deposit_id,
'metadata': expected_metadata,
'synthetic': True,
'type': 'tar'
}
expected_meta = {
'branch_name': 'master',
'origin': expected_origin,
'origin_metadata': expected_origin_metadata,
'revision': expected_revision,
}
self.assertEqual(data, expected_meta)
def test_read_metadata_5(self):
"""dateCreated/datePublished provided, revision uses author/committer
date
If multiple dateCreated provided, the first occurrence (of
dateCreated) is selected. If multiple datePublished provided,
the first occurrence (of datePublished) is selected.
"""
# add metadata to the deposit with datePublished and dateCreated
codemeta_entry_data = b"""<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom"
xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0">
<title>Composing a Web of Audio Applications</title>
<client>hal</client>
<id>hal-01243065</id>
<external_identifier>hal-01243065</external_identifier>
<codemeta:url>https://hal-test.archives-ouvertes.fr/hal-01243065</codemeta:url>
<codemeta:applicationCategory>test</codemeta:applicationCategory>
<codemeta:keywords>DSP programming,Web</codemeta:keywords>
<codemeta:dateCreated>2015-04-06T17:08:47+02:00</codemeta:dateCreated>
<codemeta:dateCreated>2016-04-06T17:08:47+02:00</codemeta:dateCreated>
<codemeta:description>this is the description</codemeta:description>
<codemeta:version>1</codemeta:version>
<codemeta:runtimePlatform>phpstorm</codemeta:runtimePlatform>
<codemeta:developmentStatus>stable</codemeta:developmentStatus>
<codemeta:programmingLanguage>php</codemeta:programmingLanguage>
<codemeta:programmingLanguage>python</codemeta:programmingLanguage>
<codemeta:programmingLanguage>C</codemeta:programmingLanguage>
<codemeta:datePublished>2017-05-03T16:08:47+02:00</codemeta:datePublished>
<codemeta:datePublished>2018-05-03T16:08:47+02:00</codemeta:datePublished>
<codemeta:license>
<codemeta:name>GNU General Public License v3.0 only</codemeta:name>
</codemeta:license>
<codemeta:license>
<codemeta:name>CeCILL Free Software License Agreement v1.1</codemeta:name>
</codemeta:license>
<author>
<name>HAL</name>
<email>hal@ccsd.cnrs.fr</email>
</author>
<codemeta:author>
<codemeta:name>Morane Gruenpeter</codemeta:name>
</codemeta:author>
</entry>""" # noqa
deposit_id = self.create_deposit_partial_with_data_in_args(
codemeta_entry_data)
url = reverse(PRIVATE_GET_DEPOSIT_METADATA,
args=[self.collection.name, deposit_id])
response = self.client.get(url)
self.assertEqual(response.status_code,
status.HTTP_200_OK)
self.assertEqual(response._headers['content-type'][1],
'application/json')
data = response.json()
expected_origin = {
'type': 'deposit',
'url': 'https://hal-test.archives-ouvertes.fr/hal-01243065'
}
expected_metadata = {
'@xmlns': 'http://www.w3.org/2005/Atom',
'@xmlns:codemeta':
'https://doi.org/10.5063/SCHEMA/CODEMETA-2.0',
'author': {
'email': 'hal@ccsd.cnrs.fr',
'name': 'HAL'
},
'client': 'hal',
'codemeta:applicationCategory': 'test',
'codemeta:author': {
'codemeta:name': 'Morane Gruenpeter'
},
'codemeta:dateCreated': [
'2015-04-06T17:08:47+02:00',
'2016-04-06T17:08:47+02:00',
],
'codemeta:datePublished': [
'2017-05-03T16:08:47+02:00',
'2018-05-03T16:08:47+02:00',
],
'codemeta:description': 'this is the description',
'codemeta:developmentStatus': 'stable',
'codemeta:keywords': 'DSP programming,Web',
'codemeta:license': [
{
'codemeta:name': 'GNU General Public License v3.0 only'
},
{
'codemeta:name':
'CeCILL Free Software License Agreement v1.1'
}
],
'codemeta:programmingLanguage': [
'php', 'python', 'C'
],
'codemeta:runtimePlatform': 'phpstorm',
'codemeta:url': 'https://hal-test.archives-ouvertes.fr/hal-01243065', # noqa
'codemeta:version': '1',
'external_identifier': 'hal-01243065',
'id': 'hal-01243065',
'title': 'Composing a Web of Audio Applications'
}
expected_origin_metadata = {
'metadata': expected_metadata,
'provider': {
'metadata': {},
'provider_name': 'hal',
'provider_type': 'deposit_client',
'provider_url': 'https://hal-test.archives-ouvertes.fr/'
},
'tool': {
'configuration': {
'sword_version': '2'
},
'name': 'swh-deposit',
'version': '0.0.1'
}
}
expected_revision = {
'author': {
'email': 'robot@softwareheritage.org',
'fullname': 'Software Heritage',
'name': 'Software Heritage'
},
'committer': {
'email': 'robot@softwareheritage.org',
'fullname': 'Software Heritage',
'name': 'Software Heritage'
},
'committer_date': {
'negative_utc': False,
'offset': 120,
'timestamp': {
'microseconds': 0,
'seconds': 1493820527
}
},
'date': {
'negative_utc': False,
'offset': 120,
'timestamp': {
'microseconds': 0,
'seconds': 1428332927
}
},
'message': 'hal: Deposit %s in collection hal' % deposit_id,
'metadata': expected_metadata,
'synthetic': True,
'type': 'tar'
}
expected_meta = {
'branch_name': 'master',
'origin': expected_origin,
'origin_metadata': expected_origin_metadata,
'revision': expected_revision,
}
self.assertEqual(data, expected_meta)
def test_access_to_nonexisting_deposit_returns_404_response(self): def test_access_to_nonexisting_deposit_returns_404_response(self):
"""Read unknown collection should return a 404 response """Read unknown collection should return a 404 response
""" """
unknown_id = '999' unknown_id = '999'
url = reverse(PRIVATE_GET_DEPOSIT_METADATA, url = reverse(PRIVATE_GET_DEPOSIT_METADATA,
args=[self.collection.name, unknown_id]) args=[self.collection.name, unknown_id])
Show All 20 Lines
View Options

swh/deposit/tests/common.py

Show First 20 Lines • Show All 340 Lines • ▼ Show 20 Lines def setUp(self):
<url>https://hal-test.archives-ouvertes.fr/some-external-id</url> <url>https://hal-test.archives-ouvertes.fr/some-external-id</url>
<author>some awesome author</author> <author>some awesome author</author>
</entry>""" </entry>"""
self.atom_entry_data1 = b"""<?xml version="1.0"?> self.atom_entry_data1 = b"""<?xml version="1.0"?>
<entry xmlns="http://www.w3.org/2005/Atom"> <entry xmlns="http://www.w3.org/2005/Atom">
<author>another one</author> <author>another one</author>
<author>no one</author> <author>no one</author>
<codemeta:dateCreated>2017-10-07T15:17:08Z</codemeta:dateCreated>
</entry>""" </entry>"""
self.atom_entry_data2 = b"""<?xml version="1.0"?> self.atom_entry_data2 = b"""<?xml version="1.0"?>
<entry xmlns="http://www.w3.org/2005/Atom"> <entry xmlns="http://www.w3.org/2005/Atom">
<title>Awesome Compiler</title> <title>Awesome Compiler</title>
<id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id> <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
<external_identifier>1785io25c695</external_identifier> <external_identifier>1785io25c695</external_identifier>
<updated>2017-10-07T15:17:08Z</updated> <updated>2017-10-07T15:17:08Z</updated>
▲ Show 20 Lines • Show All 208 Lines • Show Last 20 Lines
View Options

swh/deposit/tests/loader/test_loader.py

Show First 20 Lines • Show All 53 Lines • ▼ Show 20 Linesclass DepositLoaderScenarioTest(APITestCase, WithAuthTestCase,
BaseLoaderStorageTest): BaseLoaderStorageTest):
def setUp(self): def setUp(self):
super().setUp() super().setUp()
# create the extraction dir used by the loader # create the extraction dir used by the loader
os.makedirs(TEST_LOADER_CONFIG['extraction_dir'], exist_ok=True) os.makedirs(TEST_LOADER_CONFIG['extraction_dir'], exist_ok=True)
# 1. create a deposit with archive and metadata # Sets a basic client which accesses the test data
self.deposit_id = self.create_simple_binary_deposit()
# 2. Sets a basic client which accesses the test data
loader_client = SWHDepositTestClient(self.client, loader_client = SWHDepositTestClient(self.client,
config=CLIENT_TEST_CONFIG) config=CLIENT_TEST_CONFIG)
# 3. setup loader with that client # Setup loader with that client
self.loader = loader.DepositLoader(client=loader_client) self.loader = loader.DepositLoader(client=loader_client)
self.storage = self.loader.storage self.storage = self.loader.storage
def tearDown(self): def tearDown(self):
super().tearDown() super().tearDown()
shutil.rmtree(TEST_LOADER_CONFIG['extraction_dir']) shutil.rmtree(TEST_LOADER_CONFIG['extraction_dir'])
def test_inject_deposit_ready(self): def test_inject_deposit_ready(self):
"""Load a deposit which is ready """Load a deposit which is ready
""" """
args = [self.collection.name, self.deposit_id] # create a deposit with archive and metadata
deposit_id = self.create_simple_binary_deposit()
ardumontAuthorUnsubmitted
Done
Inline Actions

This was actually wrong to trigger a load of a deposit with a wrong status (partial here which is only possible in tests).

ardumont: This was actually wrong to trigger a load of a deposit with a wrong status (partial here which…
ardumontAuthorUnsubmitted
Done
Inline Actions

That's why i changed the test ;)

ardumont: That's why i changed the test ;)
self.update_binary_deposit(deposit_id, status_partial=False)
args = [self.collection.name, deposit_id]
archive_url = reverse(PRIVATE_GET_RAW_CONTENT, args=args) archive_url = reverse(PRIVATE_GET_RAW_CONTENT, args=args)
deposit_meta_url = reverse(PRIVATE_GET_DEPOSIT_METADATA, args=args) deposit_meta_url = reverse(PRIVATE_GET_DEPOSIT_METADATA, args=args)
deposit_update_url = reverse(PRIVATE_PUT_DEPOSIT, args=args) deposit_update_url = reverse(PRIVATE_PUT_DEPOSIT, args=args)
# when # when
res = self.loader.load(archive_url=archive_url, res = self.loader.load(archive_url=archive_url,
deposit_meta_url=deposit_meta_url, deposit_meta_url=deposit_meta_url,
deposit_update_url=deposit_update_url) deposit_update_url=deposit_update_url)
# then # then
self.assertEqual(res['status'], 'eventful', res) self.assertEqual(res['status'], 'eventful', res)
self.assertCountContents(1) self.assertCountContents(1)
self.assertCountDirectories(1) self.assertCountDirectories(1)
self.assertCountRevisions(1) self.assertCountRevisions(1)
self.assertCountReleases(0) self.assertCountReleases(0)
self.assertCountSnapshots(1) self.assertCountSnapshots(1)
def test_inject_deposit_verify_metadata(self): def test_inject_deposit_verify_metadata(self):
"""Load a deposit with metadata, test metadata integrity """Load a deposit with metadata, test metadata integrity
""" """
self.deposit_metadata_id = self.add_metadata_to_deposit( deposit_id = self.create_simple_binary_deposit()
self.deposit_id) self.add_metadata_to_deposit(deposit_id, status_partial=False)
args = [self.collection.name, self.deposit_metadata_id] args = [self.collection.name, deposit_id]
archive_url = reverse(PRIVATE_GET_RAW_CONTENT, args=args) archive_url = reverse(PRIVATE_GET_RAW_CONTENT, args=args)
deposit_meta_url = reverse(PRIVATE_GET_DEPOSIT_METADATA, args=args) deposit_meta_url = reverse(PRIVATE_GET_DEPOSIT_METADATA, args=args)
deposit_update_url = reverse(PRIVATE_PUT_DEPOSIT, args=args) deposit_update_url = reverse(PRIVATE_PUT_DEPOSIT, args=args)
# when # when
self.loader.load(archive_url=archive_url, self.loader.load(archive_url=archive_url,
deposit_meta_url=deposit_meta_url, deposit_meta_url=deposit_meta_url,
Show All 38 Lines def test_inject_deposit_verify_metadata(self):
'id': 'hal-01243065', 'id': 'hal-01243065',
'client': 'hal', 'client': 'hal',
codemeta + 'keywords': 'DSP programming,Web', codemeta + 'keywords': 'DSP programming,Web',
codemeta + 'developmentStatus': 'stable' codemeta + 'developmentStatus': 'stable'
} }
self.assertOriginMetadataContains('deposit', origin_url, self.assertOriginMetadataContains('deposit', origin_url,
expected_origin_metadata) expected_origin_metadata)
deposit = Deposit.objects.get(pk=self.deposit_id) deposit = Deposit.objects.get(pk=deposit_id)
self.assertRegex(deposit.swh_id, r'^swh:1:dir:.*') self.assertRegex(deposit.swh_id, r'^swh:1:dir:.*')
self.assertEqual(deposit.swh_id_context, '%s;origin=%s' % ( self.assertEqual(deposit.swh_id_context, '%s;origin=%s' % (
deposit.swh_id, origin_url deposit.swh_id, origin_url
)) ))
self.assertRegex(deposit.swh_anchor_id, r'^swh:1:rev:.*') self.assertRegex(deposit.swh_anchor_id, r'^swh:1:rev:.*')
self.assertEqual(deposit.swh_anchor_id_context, '%s;origin=%s' % ( self.assertEqual(deposit.swh_anchor_id_context, '%s;origin=%s' % (
deposit.swh_anchor_id, origin_url deposit.swh_anchor_id, origin_url
)) ))
About · Developer information · Legal