Page MenuHomeSoftware Heritage

Fix XML parsing of the client.
ClosedPublic

Authored by vlorentz on Thu, Nov 19, 2:50 PM.

Details

Summary

It was intentionally namespace-unaware, but for bad reasons.

This commit also updates the mocked XML receipts, to what the server
would return now (ie. tags in the right namespace, in addition to the
deprecated ones.).

Diff Detail

Repository
rDDEP Push deposit
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Thu, Nov 19, 2:50 PM

Build has FAILED

Patch application report for D4529 (id=16043)

Could not rebase; Attempt merge onto 85416fb165...

Updating 85416fb1..b0bf5a8a
Fast-forward
 swh/deposit/cli/client.py                          |  4 +-
 swh/deposit/client.py                              | 85 +++++++---------------
 swh/deposit/parsers.py                             |  3 +-
 swh/deposit/templates/deposit/content.xml          | 23 +++---
 swh/deposit/templates/deposit/deposit_receipt.xml  | 13 +++-
 swh/deposit/templates/deposit/error.xml            |  2 +-
 swh/deposit/templates/deposit/status.xml           | 18 ++++-
 swh/deposit/tests/api/conftest.py                  |  2 +-
 swh/deposit/tests/api/test_collection_post_atom.py | 12 +--
 .../tests/api/test_collection_post_binary.py       | 18 +++--
 .../tests/api/test_collection_post_metadata.py     |  4 +-
 .../tests/api/test_collection_post_multipart.py    |  6 +-
 .../tests/api/test_deposit_private_check.py        |  2 +-
 swh/deposit/tests/api/test_deposit_schedule.py     |  2 +-
 swh/deposit/tests/api/test_deposit_status.py       | 30 ++++----
 swh/deposit/tests/api/test_get_file.py             | 10 +--
 swh/deposit/tests/cli/test_client.py               | 34 ++++++---
 swh/deposit/tests/conftest.py                      |  2 +-
 .../tests/data/https_deposit.swh.test/1_test       | 10 ++-
 .../tests/data/https_deposit.test.metadata/1_test  | 12 ++-
 .../1_test_666_metadata                            | 11 ++-
 .../https_deposit.test.metadata/1_test_666_status  | 10 ++-
 .../https_deposit.test.status/1_test_1033_status   | 12 ++-
 .../1_test_123_metadata                            | 12 ++-
 .../1_test_123_status                              | 13 +++-
 .../1_test_321_status                              | 10 ++-
 swh/deposit/urls.py                                |  2 +-
 27 files changed, 220 insertions(+), 142 deletions(-)
Changes applied before test
commit b0bf5a8aa2882246d5cc38bfbbc1aa82a34b04b7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 14:50:24 2020 +0100

    Fix XML parsing of the client.
    
    It was intentionally namespace-unaware, but for bad reasons.
    
    This commit also updates the mocked XML receipts, to what the server
    would return now (ie. tags in the right namespace, in addition to the
    deprecated ones.).

commit af8ced8f06844e4ad07ccc6ec0b11bf7d12fe86d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 14:45:21 2020 +0100

    Fix SWORD XMLNS (http://purl.org/net/sword/ -> http://purl.org/net/sword/terms/)

commit 83f46bbebb78667b4596b42aad074e47ffc6ccd1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 14:00:10 2020 +0100

    Remove useless 'type: ignore' annotation.
    
    mypy complains about it.

commit ca3422a27365c543db76db003cc2c82b329dc29b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 13:59:37 2020 +0100

    Move SWH-specific tags to the https://www.softwareheritage.org/schema/2018/deposit namespace
    
    They don't belong in SWORD.
    
    We're keeping the old ones for existing clients, and we will have to
    notify them of the change before removing the old tags.

commit 918d39057abe4a9194b0f7a0cc04e39bd297ec13
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 13:56:39 2020 +0100

    Remove <sword:request>.
    
    Motivation:
    
    1. it is not a SWORD tag
    2. I don't think there is any use for this
    3. Its content is buggy, as it only handles one level of hierarchy for the metadata values.

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/389/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/389/console

Build is green

Patch application report for D4529 (id=16053)

Could not rebase; Attempt merge onto 85416fb165...

Updating 85416fb1..6939c392
Fast-forward
 swh/deposit/cli/client.py                          |  4 +-
 swh/deposit/client.py                              | 85 +++++++---------------
 swh/deposit/parsers.py                             |  3 +-
 swh/deposit/templates/deposit/content.xml          | 23 +++---
 swh/deposit/templates/deposit/deposit_receipt.xml  | 13 +++-
 swh/deposit/templates/deposit/error.xml            |  2 +-
 swh/deposit/templates/deposit/status.xml           | 18 ++++-
 swh/deposit/tests/api/conftest.py                  |  2 +-
 swh/deposit/tests/api/test_collection.py           |  6 +-
 swh/deposit/tests/api/test_collection_post_atom.py | 12 +--
 .../tests/api/test_collection_post_binary.py       | 18 +++--
 .../tests/api/test_collection_post_metadata.py     |  4 +-
 .../tests/api/test_collection_post_multipart.py    |  6 +-
 .../tests/api/test_deposit_private_check.py        |  4 +-
 swh/deposit/tests/api/test_deposit_schedule.py     |  4 +-
 swh/deposit/tests/api/test_deposit_status.py       | 30 ++++----
 swh/deposit/tests/api/test_get_file.py             | 10 +--
 swh/deposit/tests/cli/test_client.py               | 36 +++++----
 swh/deposit/tests/conftest.py                      |  2 +-
 .../tests/data/https_deposit.swh.test/1_test       | 10 ++-
 .../tests/data/https_deposit.test.metadata/1_test  | 12 ++-
 .../1_test_666_metadata                            | 11 ++-
 .../https_deposit.test.metadata/1_test_666_status  | 10 ++-
 .../https_deposit.test.status/1_test_1033_status   | 12 ++-
 .../1_test_123_metadata                            | 12 ++-
 .../1_test_123_status                              | 13 +++-
 .../1_test_321_status                              | 10 ++-
 27 files changed, 225 insertions(+), 147 deletions(-)
Changes applied before test
commit 6939c39270a95c5afe7cce31c977da180c5eeb6c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 14:50:24 2020 +0100

    Fix XML parsing of the client.
    
    It was intentionally namespace-unaware, but for bad reasons.
    
    This commit also updates the mocked XML receipts, to what the server
    would return now (ie. tags in the right namespace, in addition to the
    deprecated ones.).

commit b56b1eff6083759dab3d3e67701aff294557804e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 14:45:21 2020 +0100

    Fix SWORD XMLNS (http://purl.org/net/sword/ -> http://purl.org/net/sword/terms/)

commit 36f75b434cfe6954f9959cda0180f6807b96e89f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 13:59:37 2020 +0100

    Move SWH-specific tags to the https://www.softwareheritage.org/schema/2018/deposit namespace
    
    They don't belong in SWORD.
    
    We're keeping the old ones for existing clients, and we will have to
    notify them of the change before removing the old tags.

commit 918d39057abe4a9194b0f7a0cc04e39bd297ec13
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 13:56:39 2020 +0100

    Remove <sword:request>.
    
    Motivation:
    
    1. it is not a SWORD tag
    2. I don't think there is any use for this
    3. Its content is buggy, as it only handles one level of hierarchy for the metadata values.

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/393/ for more details.

ardumont added inline comments.
swh/deposit/client.py
92–93

careful with this, i think that requires the DJANGO_SETTING_MODULE.... (or something) variable within the env for that to work.

Thus why, that's somehow duplicated here.

vlorentz added inline comments.Thu, Nov 19, 5:10 PM
swh/deposit/client.py
92–93

I don't follow. That's client code, what does Django have to do with it?

Build is green

Patch application report for D4529 (id=16100)

Could not rebase; Attempt merge onto cb733dd49a...

Updating cb733dd4..6fad7765
Fast-forward
 docs/endpoints/collection.rst                      | 17 +++--
 docs/endpoints/content.rst                         | 43 +++--------
 docs/endpoints/status.rst                          | 28 ++++---
 swh/deposit/cli/client.py                          |  4 +-
 swh/deposit/client.py                              | 85 +++++++---------------
 swh/deposit/parsers.py                             |  3 +-
 swh/deposit/templates/deposit/content.xml          | 23 +++---
 swh/deposit/templates/deposit/deposit_receipt.xml  | 13 +++-
 swh/deposit/templates/deposit/error.xml            |  2 +-
 swh/deposit/templates/deposit/status.xml           | 18 ++++-
 swh/deposit/tests/api/conftest.py                  |  2 +-
 swh/deposit/tests/api/test_collection.py           |  6 +-
 swh/deposit/tests/api/test_collection_post_atom.py | 12 +--
 .../tests/api/test_collection_post_binary.py       | 18 +++--
 .../tests/api/test_collection_post_metadata.py     |  4 +-
 .../tests/api/test_collection_post_multipart.py    |  6 +-
 .../tests/api/test_deposit_private_check.py        |  4 +-
 swh/deposit/tests/api/test_deposit_schedule.py     |  4 +-
 swh/deposit/tests/api/test_deposit_status.py       | 30 ++++----
 swh/deposit/tests/api/test_get_file.py             | 10 +--
 swh/deposit/tests/cli/test_client.py               | 41 ++++++-----
 swh/deposit/tests/conftest.py                      |  2 +-
 .../tests/data/https_deposit.swh.test/1_test       | 10 ++-
 .../tests/data/https_deposit.test.metadata/1_test  | 12 ++-
 .../1_test_666_metadata                            | 11 ++-
 .../https_deposit.test.metadata/1_test_666_status  | 10 ++-
 .../https_deposit.test.status/1_test_1033_status   | 12 ++-
 .../1_test_123_metadata                            | 12 ++-
 .../1_test_123_status                              | 13 +++-
 .../1_test_321_status                              | 10 ++-
 30 files changed, 266 insertions(+), 199 deletions(-)
Changes applied before test
commit 6fad77653eab1738c97d1df21489178bc1f02022
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 14:50:24 2020 +0100

    Fix XML parsing of the client.
    
    It was intentionally namespace-unaware, but for bad reasons.
    
    This commit also updates the mocked XML receipts, to what the server
    would return now (ie. tags in the right namespace, in addition to the
    deprecated ones.).

commit 0d7948b2f393f9f3145bb8be34b81313d4c8d86a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 14:45:21 2020 +0100

    Fix SWORD XMLNS (http://purl.org/net/sword/ -> http://purl.org/net/sword/terms/)

commit 0ef9273dd85904185116d123fdc55be2a3149dc0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 13:59:37 2020 +0100

    Move SWH-specific tags to the https://www.softwareheritage.org/schema/2018/deposit namespace
    
    They don't belong in SWORD.
    
    We're keeping the old ones for existing clients, and we will have to
    notify them of the change before removing the old tags.

commit 8ff2e68d8c32ebed6b11f062f643612e6a569725
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 13:56:39 2020 +0100

    Remove <sword:request>.
    
    Motivation:
    
    1. it is not a SWORD tag
    2. I don't think there is any use for this
    3. Its content is buggy, as it only handles one level of hierarchy for the metadata values.

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/402/ for more details.

douardda accepted this revision.Fri, Nov 20, 3:06 PM
douardda added a subscriber: douardda.

it looks good to me, but, yeah...

This revision is now accepted and ready to land.Fri, Nov 20, 3:06 PM
ardumont added inline comments.Fri, Nov 20, 3:11 PM
swh/deposit/client.py
92–93

I recall the code you are adding was the one I initially opened the client.

But i had issues at the time, i recall it would not work because it required to have the env variable i mentioned.

from rest_framework.parsers import BaseParser, FileUploadParser, MultiPartParser

class SWHXMLParser(BaseParser)
   ....

rest_framework pulls django deps iirc ¯\_(ツ)_/¯

It's probably fine with tests since everything is configured for it, django wise...

I'm just saying, have you checked the actual runtime with those changes?

vlorentz added inline comments.Fri, Nov 20, 3:14 PM
swh/deposit/client.py
92–93

oooh, ok, you're right.

for future readers: the rebase fixes the dependency on django thanks to D4547

Build is green

Patch application report for D4529 (id=16129)

Could not rebase; Attempt merge onto f4b4c25e44...

Updating f4b4c25e..0bb516d7
Fast-forward
 docs/endpoints/collection.rst                      | 17 ++--
 docs/endpoints/content.rst                         | 43 +++-------
 docs/endpoints/status.rst                          | 28 ++++---
 swh/deposit/cli/client.py                          |  4 +-
 swh/deposit/client.py                              | 96 ++++++----------------
 swh/deposit/templates/deposit/content.xml          | 23 +++---
 swh/deposit/templates/deposit/deposit_receipt.xml  | 13 ++-
 swh/deposit/templates/deposit/error.xml            |  2 +-
 swh/deposit/templates/deposit/status.xml           | 18 +++-
 swh/deposit/tests/api/conftest.py                  |  2 +-
 swh/deposit/tests/api/test_collection.py           |  6 +-
 swh/deposit/tests/api/test_collection_post_atom.py | 12 +--
 .../tests/api/test_collection_post_binary.py       | 18 ++--
 .../tests/api/test_collection_post_metadata.py     |  4 +-
 .../tests/api/test_collection_post_multipart.py    |  6 +-
 .../tests/api/test_deposit_private_check.py        |  4 +-
 swh/deposit/tests/api/test_deposit_schedule.py     |  4 +-
 swh/deposit/tests/api/test_deposit_status.py       | 30 ++++---
 swh/deposit/tests/api/test_get_file.py             | 10 +--
 swh/deposit/tests/cli/test_client.py               | 41 ++++-----
 swh/deposit/tests/conftest.py                      |  2 +-
 .../tests/data/https_deposit.swh.test/1_test       | 10 ++-
 .../tests/data/https_deposit.test.metadata/1_test  | 12 ++-
 .../1_test_666_metadata                            | 11 ++-
 .../https_deposit.test.metadata/1_test_666_status  | 10 ++-
 .../https_deposit.test.status/1_test_1033_status   | 12 ++-
 .../1_test_123_metadata                            | 12 ++-
 .../1_test_123_status                              | 13 ++-
 .../1_test_321_status                              | 10 ++-
 swh/deposit/utils.py                               |  3 +-
 30 files changed, 265 insertions(+), 211 deletions(-)
Changes applied before test
commit 0bb516d74153959c3ac8e3308edbf38b9ce50da2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 14:50:24 2020 +0100

    Fix XML parsing of the client.
    
    It was intentionally namespace-unaware, but for bad reasons.
    
    This commit also updates the mocked XML receipts, to what the server
    would return now (ie. tags in the right namespace, in addition to the
    deprecated ones.).

commit aa7e2b1c743d3b587bc48fa57e3df3b5962d5214
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 14:45:21 2020 +0100

    Fix SWORD XMLNS (http://purl.org/net/sword/ -> http://purl.org/net/sword/terms/)

commit 794402c0f52a312bf53c757b650795f0ceb0e50a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 13:59:37 2020 +0100

    Move SWH-specific tags to the https://www.softwareheritage.org/schema/2018/deposit namespace
    
    They don't belong in SWORD.
    
    We're keeping the old ones for existing clients, and we will have to
    notify them of the change before removing the old tags.

commit 405b577f42efdb678aefa044b10599c7d2a1e97b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Nov 19 13:56:39 2020 +0100

    Remove <sword:request>.
    
    Motivation:
    
    1. it is not a SWORD tag
    2. I don't think there is any use for this
    3. Its content is buggy, as it only handles one level of hierarchy for the metadata values.

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/412/ for more details.

This revision was automatically updated to reflect the committed changes.