Page MenuHomeSoftware Heritage
Differential D4494

metadata-only: Relax mandatory constraint on slug
ClosedPublic
Actions

Authored by ardumont on Nov 17 2020, 4:13 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2537: Extend new deposit endpoint to support metadata-only deposits
Commits
rDDEPa67ed6b06b9f: deposit: Relax mandatory constraint on slug for metadata-only deposit
Summary

So it's not necessary to provide the slug for a metadata-only deposit.

Related to T2537

Test Plan

tox

Diff Detail

Repository
rDDEP Push deposit
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Nov 17 2020, 4:13 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 17 2020, 4:13 PM
swh-public-ci added a comment.Nov 17 2020, 4:16 PM

Build is green

Patch application report for D4494 (id=15940)

Could not rebase; Attempt merge onto c1a45162d3...

Updating c1a45162..efde0f59
Fast-forward
 swh/deposit/api/common.py                          | 190 ++++++++++++++-
 swh/deposit/api/deposit.py                         |  24 +-
 swh/deposit/api/deposit_update.py                  |  95 +-------
 swh/deposit/config.py                              |   5 +
 swh/deposit/parsers.py                             |  22 ++
 swh/deposit/tests/api/test_deposit_metadata.py     | 254 +++++++++++++++++++++
 swh/deposit/tests/api/test_parsers.py              |  19 +-
 swh/deposit/tests/conftest.py                      |   6 +-
 .../tests/data/atom/entry-data-with-origin.xml     |  13 ++
 .../tests/data/atom/entry-data-with-swhid.xml      |  13 ++
 swh/deposit/tests/test_utils.py                    |  61 ++++-
 swh/deposit/utils.py                               |  40 +++-
 12 files changed, 608 insertions(+), 134 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_deposit_metadata.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-origin.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid.xml
Changes applied before test
commit efde0f597b2af2eb2ef95ccc08d358dcea86e72a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 16:13:19 2020 +0100

    deposit: Relax mandatory constraint on slug for metadata-only deposit
    
    So it's not necessary to provide the slug for a metadata-only deposit.
    
    Related to T2537

commit be2056b943947cdec2395bbb80663a975a4e0793
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 15:28:27 2020 +0100

    Refuse SWHID use with qualifier anchor and visit targeting both a snapshot
    
    Related to T2537

commit 85dc1f7f3d910ffff66ef00dd875794997c4b7b3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 18:27:23 2020 +0100

    Adapt existing POST to a collection to allow metadata-only deposit
    
    The following endpoint is adapted to allow it:
    
    POST /1/<collection>
    HEADER: Content-type: application/atom+xml;type=entry
    And the xml provided by the client contains either:
    
    <swh:deposit>
      <swh:reference>
        <swh:object swhid="{swhid}" />
      </swh:reference>
    </swh:deposit>
    or:
    
    <swh:deposit>
      <swh:reference>
        <swh:origin url="{url}" />
      </swh:reference>
    </swh:deposit>
    
    Any invalid swhid raises a 400 bad request. If everything passes, a 201
    response with usual deposit receipt is received by the deposit user. The
    deposit row is updated with status "done", complete_date and reception_date to
    the same and current date, the swhid columns are updated as well if any.
    
    Note: This technically is reusing the existing code from the metadata update
    scenario. The current metadata update still happens the same way as before.
    
    Related to T2537

commit ce7ab89f267d84dabe032d2dcbe26f21b7e165f0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 14:32:05 2020 +0100

    metadata-only: Start deposit
    
    It does nothing right now but add the detection of the metadata-only deposit.
    And fails in case of invalid input.
    
    Related to T2537

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/342/ for more details.

Harbormaster completed remote builds in B17145: Diff 15940.Nov 17 2020, 4:16 PM
ardumont mentioned this in D4475: Adapt existing POST to a collection to allow metadata-only deposit.Nov 17 2020, 4:55 PM
ardumont added inline comments.Nov 17 2020, 5:07 PM
swh/deposit/api/common.py
204

*beurk*

ardumont updated this revision to Diff 15942.Nov 17 2020, 5:14 PM

Improve instruction

ardumont edited the summary of this revision. (Show Details)Nov 17 2020, 5:14 PM
swh-public-ci added a comment.Nov 17 2020, 5:17 PM

Build is green

Patch application report for D4494 (id=15942)

Could not rebase; Attempt merge onto c1a45162d3...

Updating c1a45162..f9f0af0a
Fast-forward
 swh/deposit/api/common.py                          | 190 +++++++++++++--
 swh/deposit/api/deposit.py                         |  24 +-
 swh/deposit/api/deposit_update.py                  |  95 +-------
 swh/deposit/config.py                              |   5 +
 swh/deposit/parsers.py                             |  22 ++
 swh/deposit/tests/api/test_deposit_metadata.py     | 254 +++++++++++++++++++++
 swh/deposit/tests/api/test_parsers.py              |  19 +-
 swh/deposit/tests/conftest.py                      |   6 +-
 .../tests/data/atom/entry-data-with-origin.xml     |  13 ++
 .../tests/data/atom/entry-data-with-swhid.xml      |  13 ++
 swh/deposit/tests/test_utils.py                    |  61 ++++-
 swh/deposit/utils.py                               |  40 +++-
 12 files changed, 606 insertions(+), 136 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_deposit_metadata.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-origin.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid.xml
Changes applied before test
commit f9f0af0a90bb5a9806995363e86dd718f2f05e94
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 16:13:19 2020 +0100

    deposit: Relax mandatory constraint on slug for metadata-only deposit
    
    So it's not necessary to provide the slug for a metadata-only deposit.
    
    Related to T2537

commit be2056b943947cdec2395bbb80663a975a4e0793
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 15:28:27 2020 +0100

    Refuse SWHID use with qualifier anchor and visit targeting both a snapshot
    
    Related to T2537

commit 85dc1f7f3d910ffff66ef00dd875794997c4b7b3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 18:27:23 2020 +0100

    Adapt existing POST to a collection to allow metadata-only deposit
    
    The following endpoint is adapted to allow it:
    
    POST /1/<collection>
    HEADER: Content-type: application/atom+xml;type=entry
    And the xml provided by the client contains either:
    
    <swh:deposit>
      <swh:reference>
        <swh:object swhid="{swhid}" />
      </swh:reference>
    </swh:deposit>
    or:
    
    <swh:deposit>
      <swh:reference>
        <swh:origin url="{url}" />
      </swh:reference>
    </swh:deposit>
    
    Any invalid swhid raises a 400 bad request. If everything passes, a 201
    response with usual deposit receipt is received by the deposit user. The
    deposit row is updated with status "done", complete_date and reception_date to
    the same and current date, the swhid columns are updated as well if any.
    
    Note: This technically is reusing the existing code from the metadata update
    scenario. The current metadata update still happens the same way as before.
    
    Related to T2537

commit ce7ab89f267d84dabe032d2dcbe26f21b7e165f0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 14:32:05 2020 +0100

    metadata-only: Start deposit
    
    It does nothing right now but add the detection of the metadata-only deposit.
    And fails in case of invalid input.
    
    Related to T2537

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/343/ for more details.

Harbormaster completed remote builds in B17147: Diff 15942.Nov 17 2020, 5:17 PM
vlorentz requested changes to this revision.Nov 17 2020, 5:41 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/deposit/api/common.py
206

external_id=external_id or ""

385

s/check_slug/check_slug_is_present/

460

s/ in request//, that's redundant with the error code

501

same

745

same

swh/deposit/api/deposit.py
37–50

why did you remove this function and duplicate its code?

This revision now requires changes to proceed.Nov 17 2020, 5:41 PM
ardumont added inline comments.Nov 17 2020, 6:00 PM
swh/deposit/api/common.py
206

lolsob, that just won't sink in... T.T

swh/deposit/api/deposit.py
37–50

It's not per say the same.

This one is actually triggered too soon in the current workflow we want.
This method is overridable and called from the checks method way ahead
of time before we can check for the xml data for a swhid.

So removing it and calling an equivalent (granted duplicated code [1]) code only when we actually want to check for the slug.

[1] I can probably improve on the duplication part.

vlorentz added inline comments.Nov 17 2020, 6:05 PM
swh/deposit/api/deposit.py
37–50

yeah, please deduplicate

ardumont added inline comments.Nov 17 2020, 6:11 PM
swh/deposit/api/deposit.py
37–50

I'm realizing that it will need to happen also for other errors btw (in another diff)

ardumont updated this revision to Diff 15944.Nov 17 2020, 6:11 PM

Adapt according to review

swh-public-ci added a comment.Nov 17 2020, 6:14 PM

Build is green

Patch application report for D4494 (id=15944)

Could not rebase; Attempt merge onto c1a45162d3...

Updating c1a45162..6111ab1a
Fast-forward
 swh/deposit/api/common.py                          | 179 +++++++++++++--
 swh/deposit/api/deposit.py                         |  28 +--
 swh/deposit/api/deposit_update.py                  |  95 +-------
 swh/deposit/config.py                              |   5 +
 swh/deposit/errors.py                              |  18 +-
 swh/deposit/parsers.py                             |  20 ++
 swh/deposit/tests/api/test_deposit_metadata.py     | 254 +++++++++++++++++++++
 swh/deposit/tests/api/test_parsers.py              |  21 +-
 swh/deposit/tests/conftest.py                      |   6 +-
 .../tests/data/atom/entry-data-with-origin.xml     |  13 ++
 .../tests/data/atom/entry-data-with-swhid.xml      |  13 ++
 swh/deposit/tests/test_utils.py                    |  61 ++++-
 swh/deposit/utils.py                               |  40 +++-
 13 files changed, 615 insertions(+), 138 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_deposit_metadata.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-origin.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid.xml
Changes applied before test
commit 6111ab1a0a0ab0e72ca2004e1179b723b604940d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 16:13:19 2020 +0100

    deposit: Relax mandatory constraint on slug for metadata-only deposit
    
    So it's not necessary to provide the slug for a metadata-only deposit.
    
    Related to T2537

commit 071650b783f764d40fed12d97633a3e6e816c7ce
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 15:28:27 2020 +0100

    Refuse SWHID use with qualifier anchor and visit targeting both a snapshot
    
    Related to T2537

commit 85dc1f7f3d910ffff66ef00dd875794997c4b7b3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 18:27:23 2020 +0100

    Adapt existing POST to a collection to allow metadata-only deposit
    
    The following endpoint is adapted to allow it:
    
    POST /1/<collection>
    HEADER: Content-type: application/atom+xml;type=entry
    And the xml provided by the client contains either:
    
    <swh:deposit>
      <swh:reference>
        <swh:object swhid="{swhid}" />
      </swh:reference>
    </swh:deposit>
    or:
    
    <swh:deposit>
      <swh:reference>
        <swh:origin url="{url}" />
      </swh:reference>
    </swh:deposit>
    
    Any invalid swhid raises a 400 bad request. If everything passes, a 201
    response with usual deposit receipt is received by the deposit user. The
    deposit row is updated with status "done", complete_date and reception_date to
    the same and current date, the swhid columns are updated as well if any.
    
    Note: This technically is reusing the existing code from the metadata update
    scenario. The current metadata update still happens the same way as before.
    
    Related to T2537

commit ce7ab89f267d84dabe032d2dcbe26f21b7e165f0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 14:32:05 2020 +0100

    metadata-only: Start deposit
    
    It does nothing right now but add the detection of the metadata-only deposit.
    And fails in case of invalid input.
    
    Related to T2537

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/344/ for more details.

Harbormaster completed remote builds in B17150: Diff 15944.Nov 17 2020, 6:14 PM
vlorentz accepted this revision.Nov 17 2020, 6:15 PM
This revision is now accepted and ready to land.Nov 17 2020, 6:15 PM
ardumont updated this revision to Diff 15949.Nov 17 2020, 6:34 PM

Rebase

ardumont retitled this revision from deposit: Relax mandatory constraint on slug for metadata-only deposit to metadata-only: Relax mandatory constraint on slug.Nov 17 2020, 6:35 PM
swh-public-ci added a comment.Nov 17 2020, 6:43 PM

Build is green

Patch application report for D4494 (id=15949)

Could not rebase; Attempt merge onto c1a45162d3...

Updating c1a45162..a02b868e
Fast-forward
 swh/deposit/api/common.py                          | 181 +++++++++++++--
 swh/deposit/api/deposit.py                         |  28 +--
 swh/deposit/api/deposit_update.py                  |  92 +-------
 swh/deposit/config.py                              |   5 +
 swh/deposit/errors.py                              |  18 +-
 swh/deposit/parsers.py                             |  20 ++
 swh/deposit/tests/api/test_deposit_metadata.py     | 254 +++++++++++++++++++++
 swh/deposit/tests/api/test_parsers.py              |  21 +-
 swh/deposit/tests/conftest.py                      |   6 +-
 .../tests/data/atom/entry-data-with-origin.xml     |  13 ++
 .../tests/data/atom/entry-data-with-swhid.xml      |  13 ++
 swh/deposit/tests/test_utils.py                    |  61 ++++-
 swh/deposit/utils.py                               |  40 +++-
 13 files changed, 621 insertions(+), 131 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_deposit_metadata.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-origin.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid.xml
Changes applied before test
commit a02b868e4e354e492cc2813543adb76e4845ef79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 16:13:19 2020 +0100

    deposit: Relax mandatory constraint on slug for metadata-only deposit
    
    So it's not necessary to provide the slug for a metadata-only deposit.
    
    Related to T2537

commit 546f96f4f08200ffa48a85839b7065e2ef1f4a0c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 15:28:27 2020 +0100

    Refuse SWHID use with qualifier anchor and visit targeting both a snapshot
    
    Related to T2537

commit 406790effb7080befe2eef6b0c7734a751e16870
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 14:32:05 2020 +0100

    Adapt existing POST to a collection to allow metadata-only deposit
    
    The following endpoint is adapted to allow it:
    
    POST /1/<collection>
    HEADER: Content-type: application/atom+xml;type=entry
    And the xml provided by the client contains either:
    
    <swh:deposit>
      <swh:reference>
        <swh:object swhid="{swhid}" />
      </swh:reference>
    </swh:deposit>
    or:
    
    <swh:deposit>
      <swh:reference>
        <swh:origin url="{url}" />
      </swh:reference>
    </swh:deposit>
    
    Any invalid swhid raises a 400 bad request. If everything passes, a 201
    response with usual deposit receipt is received by the deposit user. The
    deposit row is updated with status "done", complete_date and reception_date to
    the same and current date, the swhid columns are updated as well if any.
    
    Note: Technically, this is reusing the existing code from the metadata update
    scenario. The current metadata update still happens the same way as before.
    
    Related to T2537

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/348/ for more details.

Harbormaster completed remote builds in B17155: Diff 15949.Nov 17 2020, 6:43 PM
ardumont updated this revision to Diff 15955.Nov 18 2020, 9:52 AM

Rebase

swh-public-ci added a comment.Nov 18 2020, 9:58 AM

Build is green

Patch application report for D4494 (id=15955)

Could not rebase; Attempt merge onto c1a45162d3...

Updating c1a45162..a67ed6b0
Fast-forward
 swh/deposit/api/common.py                          | 176 ++++++++++++-
 swh/deposit/api/deposit.py                         |  28 +--
 swh/deposit/api/deposit_update.py                  |  83 +------
 swh/deposit/config.py                              |   5 +
 swh/deposit/errors.py                              |  32 ++-
 swh/deposit/parsers.py                             |  20 ++
 swh/deposit/tests/api/test_deposit_metadata.py     | 273 +++++++++++++++++++++
 swh/deposit/tests/api/test_parsers.py              |  21 +-
 swh/deposit/tests/conftest.py                      |   6 +-
 .../tests/data/atom/entry-data-with-origin.xml     |  13 +
 ...-with-swhid-fail-metadata-functional-checks.xml |  12 +
 .../tests/data/atom/entry-data-with-swhid.xml      |  13 +
 swh/deposit/tests/test_utils.py                    |  61 ++++-
 swh/deposit/utils.py                               |  40 ++-
 14 files changed, 658 insertions(+), 125 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_deposit_metadata.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-origin.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid-fail-metadata-functional-checks.xml
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid.xml
Changes applied before test
commit a67ed6b06b9fd250679e108fe66364246c5f3aa2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 16:13:19 2020 +0100

    deposit: Relax mandatory constraint on slug for metadata-only deposit
    
    So it's not necessary to provide the slug for a metadata-only deposit.
    
    Related to T2537

commit 5d1932ce291cdecf78f08642ce6783845c3f616d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Nov 17 15:28:27 2020 +0100

    Refuse SWHID use with qualifier anchor and visit targeting both a snapshot
    
    Related to T2537

commit b86840cd1e2278ebfbcfdc6d57f3384f28fea96c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Nov 13 14:32:05 2020 +0100

    Adapt existing POST to a collection to allow metadata-only deposit
    
    The following endpoint is adapted to allow it:
    
    POST /1/<collection>
    HEADER: Content-type: application/atom+xml;type=entry
    And the xml provided by the client contains either:
    
    <swh:deposit>
      <swh:reference>
        <swh:object swhid="{swhid}" />
      </swh:reference>
    </swh:deposit>
    or:
    
    <swh:deposit>
      <swh:reference>
        <swh:origin url="{url}" />
      </swh:reference>
    </swh:deposit>
    
    Any invalid swhid raises a 400 bad request. If everything passes, a 201
    response with usual deposit receipt is received by the deposit user. The
    deposit row is updated with status "done", complete_date and reception_date to
    the same and current date, the swhid columns are updated as well if any.
    
    Note: Technically, this is reusing the existing code from the metadata update
    scenario. The current metadata update still happens the same way as before.
    
    Related to T2537

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/354/ for more details.

Harbormaster completed remote builds in B17161: Diff 15955.Nov 18 2020, 9:58 AM
Closed by commit rDDEPa67ed6b06b9f: deposit: Relax mandatory constraint on slug for metadata-only deposit (authored by ardumont). · Explain WhyNov 18 2020, 11:13 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDDEPa67ed6b06b9f: deposit: Relax mandatory constraint on slug for metadata-only deposit.

Revision Contents
Changeset List

PathSize
swh/
deposit/
api/
36 lines
28 lines
18 lines
tests/
api/
4 lines

Diff 15962

View Options

swh/deposit/api/common.py

Loading...
View Options

swh/deposit/api/deposit.py

Loading...
View Options

swh/deposit/errors.py

Loading...
View Options

swh/deposit/tests/api/test_deposit_metadata.py

Loading...
About · Developer information · Legal