Page MenuHomeSoftware Heritage
Differential D4493

metadata-only: Restrict swhid use of qualifier anchor and visit on snapshot
ClosedPublic
Actions

Authored by ardumont on Nov 17 2020, 3:30 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2537: Extend new deposit endpoint to support metadata-only deposits
Commits
rDDEP5d1932ce291c: Refuse SWHID use with qualifier anchor and visit targeting both a snapshot
Summary

I don't know what such SWHID would mean but technically it's a valid one (as
far as i understood the grammar [1])

Nonetheless, we currently have a technical restriction when writing extrinsic
metadata though. We can only have one snapshot reference possible there...

Let's discuss what to do then ;)

In this diff, I currently propose to log and return a bad request for such case.

That's the gist of the discussion, is that reasonable? If not, what to do instead?

[1] https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html#syntax

Related to T2537 D4475

Test Plan

tox

Diff Detail

Repository
rDDEP Push deposit
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17143
Build 26460: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 26459: arc lint + arc unit

Event Timeline

ardumont created this revision.Nov 17 2020, 3:30 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 17 2020, 3:30 PM
swh-public-ci added a comment.Nov 17 2020, 3:30 PM

Build has FAILED

Patch application report for D4493 (id=15938)

Could not rebase; Attempt merge onto c1a45162d3...

Updating c1a45162..e943e216
Fast-forward
 swh/deposit/api/common.py                          | 145 +++++++++++-
 swh/deposit/api/deposit_update.py                  |  95 +-------
 swh/deposit/config.py                              |   5 +
 swh/deposit/parsers.py                             |  12 +
 swh/deposit/tests/api/test_deposit_metadata.py     | 256 +++++++++++++++++++++
 swh/deposit/tests/api/test_parsers.py              |  19 +-
 swh/deposit/tests/conftest.py                      |   6 +-
 .../tests/data/atom/entry-data-with-origin.xml     |  13 ++
 .../tests/data/atom/entry-data-with-swhid.xml      |  13 ++
 swh/deposit/tests/test_utils.py                    |  61 ++++-
 swh/deposit/utils.py                               |  40 +++-
 11 files changed, 555 insertions(+), 110 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_deposit_metadata.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-origin.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid.xml
Changes applied before test
commit e943e2163ee7315a48fe2b251f5b125a6a5dabda
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 15:28:27 2020 +0100

    metadata-only: Restrict swhid use of qualifier anchor and visit on snapshot
    
    Related to T2537

commit 85dc1f7f3d910ffff66ef00dd875794997c4b7b3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 18:27:23 2020 +0100

    Adapt existing POST to a collection to allow metadata-only deposit
    
    The following endpoint is adapted to allow it:
    
    POST /1/<collection>
    HEADER: Content-type: application/atom+xml;type=entry
    And the xml provided by the client contains either:
    
    <swh:deposit>
      <swh:reference>
        <swh:object swhid="{swhid}" />
      </swh:reference>
    </swh:deposit>
    or:
    
    <swh:deposit>
      <swh:reference>
        <swh:origin url="{url}" />
      </swh:reference>
    </swh:deposit>
    
    Any invalid swhid raises a 400 bad request. If everything passes, a 201
    response with usual deposit receipt is received by the deposit user. The
    deposit row is updated with status "done", complete_date and reception_date to
    the same and current date, the swhid columns are updated as well if any.
    
    Note: This technically is reusing the existing code from the metadata update
    scenario. The current metadata update still happens the same way as before.
    
    Related to T2537

commit ce7ab89f267d84dabe032d2dcbe26f21b7e165f0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 14:32:05 2020 +0100

    metadata-only: Start deposit
    
    It does nothing right now but add the detection of the metadata-only deposit.
    And fails in case of invalid input.
    
    Related to T2537

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/340/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/340/console

Harbormaster failed remote builds in B17142: Diff 15938!Nov 17 2020, 3:30 PM
ardumont edited the summary of this revision. (Show Details)Nov 17 2020, 3:33 PM
ardumont updated this revision to Diff 15939.Nov 17 2020, 3:35 PM
ardumont edited the summary of this revision. (Show Details)

Fix test

ardumont mentioned this in D4475: Adapt existing POST to a collection to allow metadata-only deposit.Nov 17 2020, 3:36 PM
swh-public-ci added a comment.Nov 17 2020, 3:38 PM

Build is green

Patch application report for D4493 (id=15939)

Could not rebase; Attempt merge onto c1a45162d3...

Updating c1a45162..be2056b9
Fast-forward
 swh/deposit/api/common.py                          | 145 +++++++++++-
 swh/deposit/api/deposit_update.py                  |  95 +-------
 swh/deposit/config.py                              |   5 +
 swh/deposit/parsers.py                             |  22 ++
 swh/deposit/tests/api/test_deposit_metadata.py     | 256 +++++++++++++++++++++
 swh/deposit/tests/api/test_parsers.py              |  19 +-
 swh/deposit/tests/conftest.py                      |   6 +-
 .../tests/data/atom/entry-data-with-origin.xml     |  13 ++
 .../tests/data/atom/entry-data-with-swhid.xml      |  13 ++
 swh/deposit/tests/test_utils.py                    |  61 ++++-
 swh/deposit/utils.py                               |  40 +++-
 11 files changed, 565 insertions(+), 110 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_deposit_metadata.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-origin.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid.xml
Changes applied before test
commit be2056b943947cdec2395bbb80663a975a4e0793
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 15:28:27 2020 +0100

    Refuse SWHID use with qualifier anchor and visit targeting both a snapshot
    
    Related to T2537

commit 85dc1f7f3d910ffff66ef00dd875794997c4b7b3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 18:27:23 2020 +0100

    Adapt existing POST to a collection to allow metadata-only deposit
    
    The following endpoint is adapted to allow it:
    
    POST /1/<collection>
    HEADER: Content-type: application/atom+xml;type=entry
    And the xml provided by the client contains either:
    
    <swh:deposit>
      <swh:reference>
        <swh:object swhid="{swhid}" />
      </swh:reference>
    </swh:deposit>
    or:
    
    <swh:deposit>
      <swh:reference>
        <swh:origin url="{url}" />
      </swh:reference>
    </swh:deposit>
    
    Any invalid swhid raises a 400 bad request. If everything passes, a 201
    response with usual deposit receipt is received by the deposit user. The
    deposit row is updated with status "done", complete_date and reception_date to
    the same and current date, the swhid columns are updated as well if any.
    
    Note: This technically is reusing the existing code from the metadata update
    scenario. The current metadata update still happens the same way as before.
    
    Related to T2537

commit ce7ab89f267d84dabe032d2dcbe26f21b7e165f0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 14:32:05 2020 +0100

    metadata-only: Start deposit
    
    It does nothing right now but add the detection of the metadata-only deposit.
    And fails in case of invalid input.
    
    Related to T2537

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/341/ for more details.

Harbormaster completed remote builds in B17143: Diff 15939.Nov 17 2020, 3:38 PM
ardumont edited the summary of this revision. (Show Details)Nov 17 2020, 5:00 PM
ardumont edited the summary of this revision. (Show Details)Nov 17 2020, 5:15 PM
vlorentz requested changes to this revision.Nov 17 2020, 5:37 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/deposit/parsers.py
204–207

'anchor=swh:1:snp:' is not supported when 'visit' is also provided.

This revision now requires changes to proceed.Nov 17 2020, 5:37 PM
ardumont marked an inline comment as done.Nov 17 2020, 5:55 PM
ardumont added inline comments.
swh/deposit/parsers.py
204–207

thanks, clearer.

ardumont updated this revision to Diff 15945.Nov 17 2020, 6:12 PM
ardumont marked an inline comment as done.

Adapt according to review

vlorentz accepted this revision.Nov 17 2020, 6:14 PM
This revision is now accepted and ready to land.Nov 17 2020, 6:14 PM
swh-public-ci added a comment.Nov 17 2020, 6:17 PM

Build is green

Patch application report for D4493 (id=15945)

Could not rebase; Attempt merge onto c1a45162d3...

Updating c1a45162..071650b7
Fast-forward
 swh/deposit/api/common.py                          | 145 +++++++++++-
 swh/deposit/api/deposit_update.py                  |  95 +-------
 swh/deposit/config.py                              |   5 +
 swh/deposit/parsers.py                             |  20 ++
 swh/deposit/tests/api/test_deposit_metadata.py     | 256 +++++++++++++++++++++
 swh/deposit/tests/api/test_parsers.py              |  21 +-
 swh/deposit/tests/conftest.py                      |   6 +-
 .../tests/data/atom/entry-data-with-origin.xml     |  13 ++
 .../tests/data/atom/entry-data-with-swhid.xml      |  13 ++
 swh/deposit/tests/test_utils.py                    |  61 ++++-
 swh/deposit/utils.py                               |  40 +++-
 11 files changed, 564 insertions(+), 111 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_deposit_metadata.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-origin.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid.xml
Changes applied before test
commit 071650b783f764d40fed12d97633a3e6e816c7ce
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 15:28:27 2020 +0100

    Refuse SWHID use with qualifier anchor and visit targeting both a snapshot
    
    Related to T2537

commit 85dc1f7f3d910ffff66ef00dd875794997c4b7b3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 18:27:23 2020 +0100

    Adapt existing POST to a collection to allow metadata-only deposit
    
    The following endpoint is adapted to allow it:
    
    POST /1/<collection>
    HEADER: Content-type: application/atom+xml;type=entry
    And the xml provided by the client contains either:
    
    <swh:deposit>
      <swh:reference>
        <swh:object swhid="{swhid}" />
      </swh:reference>
    </swh:deposit>
    or:
    
    <swh:deposit>
      <swh:reference>
        <swh:origin url="{url}" />
      </swh:reference>
    </swh:deposit>
    
    Any invalid swhid raises a 400 bad request. If everything passes, a 201
    response with usual deposit receipt is received by the deposit user. The
    deposit row is updated with status "done", complete_date and reception_date to
    the same and current date, the swhid columns are updated as well if any.
    
    Note: This technically is reusing the existing code from the metadata update
    scenario. The current metadata update still happens the same way as before.
    
    Related to T2537

commit ce7ab89f267d84dabe032d2dcbe26f21b7e165f0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 14:32:05 2020 +0100

    metadata-only: Start deposit
    
    It does nothing right now but add the detection of the metadata-only deposit.
    And fails in case of invalid input.
    
    Related to T2537

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/345/ for more details.

Harbormaster completed remote builds in B17151: Diff 15945.Nov 17 2020, 6:17 PM
ardumont updated this revision to Diff 15948.Nov 17 2020, 6:34 PM

Rebase

swh-public-ci added a comment.Nov 17 2020, 6:40 PM

Build is green

Patch application report for D4493 (id=15948)

Could not rebase; Attempt merge onto c1a45162d3...

Updating c1a45162..546f96f4
Fast-forward
 swh/deposit/api/common.py                          | 147 +++++++++++-
 swh/deposit/api/deposit_update.py                  |  92 +-------
 swh/deposit/config.py                              |   5 +
 swh/deposit/parsers.py                             |  20 ++
 swh/deposit/tests/api/test_deposit_metadata.py     | 256 +++++++++++++++++++++
 swh/deposit/tests/api/test_parsers.py              |  21 +-
 swh/deposit/tests/conftest.py                      |   6 +-
 .../tests/data/atom/entry-data-with-origin.xml     |  13 ++
 .../tests/data/atom/entry-data-with-swhid.xml      |  13 ++
 swh/deposit/tests/test_utils.py                    |  61 ++++-
 swh/deposit/utils.py                               |  40 +++-
 11 files changed, 570 insertions(+), 104 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_deposit_metadata.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-origin.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid.xml
Changes applied before test
commit 546f96f4f08200ffa48a85839b7065e2ef1f4a0c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 15:28:27 2020 +0100

    Refuse SWHID use with qualifier anchor and visit targeting both a snapshot
    
    Related to T2537

commit 406790effb7080befe2eef6b0c7734a751e16870
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 14:32:05 2020 +0100

    Adapt existing POST to a collection to allow metadata-only deposit
    
    The following endpoint is adapted to allow it:
    
    POST /1/<collection>
    HEADER: Content-type: application/atom+xml;type=entry
    And the xml provided by the client contains either:
    
    <swh:deposit>
      <swh:reference>
        <swh:object swhid="{swhid}" />
      </swh:reference>
    </swh:deposit>
    or:
    
    <swh:deposit>
      <swh:reference>
        <swh:origin url="{url}" />
      </swh:reference>
    </swh:deposit>
    
    Any invalid swhid raises a 400 bad request. If everything passes, a 201
    response with usual deposit receipt is received by the deposit user. The
    deposit row is updated with status "done", complete_date and reception_date to
    the same and current date, the swhid columns are updated as well if any.
    
    Note: Technically, this is reusing the existing code from the metadata update
    scenario. The current metadata update still happens the same way as before.
    
    Related to T2537

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/347/ for more details.

Harbormaster completed remote builds in B17154: Diff 15948.Nov 17 2020, 6:40 PM
ardumont updated this revision to Diff 15954.Nov 18 2020, 9:52 AM

Rebase

swh-public-ci added a comment.Nov 18 2020, 9:55 AM

Build is green

Patch application report for D4493 (id=15954)

Could not rebase; Attempt merge onto c1a45162d3...

Updating c1a45162..5d1932ce
Fast-forward
 swh/deposit/api/common.py                          | 142 ++++++++++-
 swh/deposit/api/deposit_update.py                  |  83 +-----
 swh/deposit/config.py                              |   5 +
 swh/deposit/errors.py                              |  14 ++
 swh/deposit/parsers.py                             |  20 ++
 swh/deposit/tests/api/test_deposit_metadata.py     | 277 +++++++++++++++++++++
 swh/deposit/tests/api/test_parsers.py              |  21 +-
 swh/deposit/tests/conftest.py                      |   6 +-
 .../tests/data/atom/entry-data-with-origin.xml     |  13 +
 ...-with-swhid-fail-metadata-functional-checks.xml |  12 +
 .../tests/data/atom/entry-data-with-swhid.xml      |  13 +
 swh/deposit/tests/test_utils.py                    |  61 ++++-
 swh/deposit/utils.py                               |  40 ++-
 13 files changed, 609 insertions(+), 98 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_deposit_metadata.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-origin.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid-fail-metadata-functional-checks.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid.xml
Changes applied before test
commit 5d1932ce291cdecf78f08642ce6783845c3f616d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 15:28:27 2020 +0100

    Refuse SWHID use with qualifier anchor and visit targeting both a snapshot
    
    Related to T2537

commit b86840cd1e2278ebfbcfdc6d57f3384f28fea96c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 14:32:05 2020 +0100

    Adapt existing POST to a collection to allow metadata-only deposit
    
    The following endpoint is adapted to allow it:
    
    POST /1/<collection>
    HEADER: Content-type: application/atom+xml;type=entry
    And the xml provided by the client contains either:
    
    <swh:deposit>
      <swh:reference>
        <swh:object swhid="{swhid}" />
      </swh:reference>
    </swh:deposit>
    or:
    
    <swh:deposit>
      <swh:reference>
        <swh:origin url="{url}" />
      </swh:reference>
    </swh:deposit>
    
    Any invalid swhid raises a 400 bad request. If everything passes, a 201
    response with usual deposit receipt is received by the deposit user. The
    deposit row is updated with status "done", complete_date and reception_date to
    the same and current date, the swhid columns are updated as well if any.
    
    Note: Technically, this is reusing the existing code from the metadata update
    scenario. The current metadata update still happens the same way as before.
    
    Related to T2537

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/353/ for more details.

Harbormaster completed remote builds in B17160: Diff 15954.Nov 18 2020, 9:55 AM
Closed by commit rDDEP5d1932ce291c: Refuse SWHID use with qualifier anchor and visit targeting both a snapshot (authored by ardumont). · Explain WhyNov 18 2020, 11:13 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDDEP5d1932ce291c: Refuse SWHID use with qualifier anchor and visit targeting both a snapshot.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
deposit/
22 lines100%
Loading...
tests/
api/
4 lines100%
Loading...

Diff 15939

View Options

swh/deposit/parsers.py

Loading...
View Options

swh/deposit/tests/api/test_parsers.py

Loading...
About · Developer information · Legal