Page MenuHomeSoftware Heritage
Differential D7249

Use xmlschema to validate dates, instead of custom code.
ClosedPublic
Actions

Authored by vlorentz on Feb 24 2022, 3:59 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T3965: deposit: Validate Atom documents are well-formed
Commits
rDDEP74d5567b31f4: Use xmlschema to validate dates, instead of custom code.
Summary

For now this increases code complexity, but this will allow addition
of other check more easily.

Diff Detail

Repository
rDDEP Push deposit
Branch
validate-metadata
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 27141
Build 42453: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 42452: arc lint + arc unit

Event Timeline

vlorentz created this revision.Feb 24 2022, 3:59 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 24 2022, 3:59 PM
vlorentz added a child revision: D7250: Validate codemeta:dateModified..Feb 24 2022, 4:05 PM
vlorentz added a parent revision: D7244: Enforce create_origin/add_to_origin/reference are mutually exclusive in the schema.Feb 24 2022, 4:05 PM
swh-public-ci added a comment.Feb 24 2022, 4:06 PM

Build is green

Patch application report for D7249 (id=26270)

Could not rebase; Attempt merge onto ccff46762b...

Removing docs/specs/swh.xsd
Merge made by the 'recursive' strategy.
 MANIFEST.in                                        |   1 +
 docs/specs/protocol-reference.rst                  |   4 +-
 docs/specs/swh.xsd                                 |  60 ----
 mypy.ini                                           |   3 +
 requirements-server.txt                            |   1 +
 swh/deposit/api/checks.py                          |  59 +++-
 swh/deposit/tests/api/test_checks.py               | 371 ++++++++++++++++++---
 .../{tests_migration => tests}/test_migrations.py  |   0
 swh/deposit/xsd/codemeta.xsd                       |  16 +
 swh/deposit/xsd/swh.xsd                            |  81 +++++
 10 files changed, 473 insertions(+), 123 deletions(-)
 delete mode 100644 docs/specs/swh.xsd
 rename swh/deposit/{tests_migration => tests}/test_migrations.py (100%)
 create mode 100644 swh/deposit/xsd/codemeta.xsd
 create mode 100644 swh/deposit/xsd/swh.xsd
Changes applied before test
commit c4e82931ca0972a5762cf494a2f874f0baef284f
Merge: ccff4676 d9b748d7
Author: Jenkins user <jenkins@localhost>
Date:   Thu Feb 24 14:59:29 2022 +0000

    Merge branch 'diff-target' into HEAD

commit d9b748d7d4779959e8f397526016ecd496dbcc44
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 15:57:40 2022 +0100

    Use xmlschema to validate dates, instead of custom code.
    
    For now this increases code complexity, but this will allow addition
    of other check more easily.

commit fb205f30fc92eb35631705cdb6ec9b639f6fe6b1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 13:35:42 2022 +0100

    Enforce create_origin/add_to_origin/reference are mutually exclusive in the schema

commit a0d5f3a6c4800f6ee460b30ad6eee04caf523e89
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 13:30:33 2022 +0100

    Add schema validation of <swh:deposit> using swh.xsd

commit 89c086661371c7dad7b661fb967b451c41b5d955
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 13:24:10 2022 +0100

    Improve readability of pytest errors by removing unnecessary whitespaces
    
    The leading newline prevented textwrap.dedent from removing them.

commit d2d904a283a1434d30cf8c68bfc9e708753f705f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 12:13:50 2022 +0100

    Fix swh.xsd
    
    It was never actually tested...

commit d56f08cb83bd25c70a8cb4bafc6df71b6844f5d2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 12:04:03 2022 +0100

    Move tests_migration/test_migrations.py to tests/test_migrations.py
    
    This fixes crashes when running 'pytest -k migration', because swh/deposit/tests_migration/
    lacks a conftest to initialize the database.

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/770/ for more details.

Harbormaster completed remote builds in B27141: Diff 26270.Feb 24 2022, 4:06 PM
vlorentz requested review of this revision.Feb 24 2022, 4:06 PM
vlorentz added a task: T3965: deposit: Validate Atom documents are well-formed.Feb 24 2022, 4:32 PM
ardumont added a subscriber: ardumont.Feb 24 2022, 5:46 PM
ardumont added inline comments.
swh/deposit/api/checks.py
110

you dropped the suggested field missing, is that intended?

vlorentz added inline comments.Feb 24 2022, 5:57 PM
swh/deposit/api/checks.py
110

hmm it wasn't, but it's annoying to test when it is there

ardumont added inline comments.Feb 24 2022, 6:01 PM
swh/deposit/api/checks.py
110

is it? there was a test about it before, what changed?

I'd prefer we leave it though, so at least, when someone provides something wrong and without the suggested field, it sees the suggestion nonetheless and fix it along the rest?

ardumont accepted this revision.Feb 24 2022, 6:40 PM

lgtm

I'd like the part dropped to stay though (if possible ;)

This revision is now accepted and ready to land.Feb 24 2022, 6:40 PM
vlorentz added a comment.EditedFeb 28 2022, 10:17 AM

Why?

This will be dead code if we keep it.

vlorentz added a comment.Feb 28 2022, 10:22 AM

Oh I misunderstood what part you were referring to. I'll send another diff to restore it.

vlorentz updated this revision to Diff 26304.Feb 28 2022, 10:23 AM

rebase

vlorentz mentioned this in D7263: Warn about missing <swh:metadata-provenance> even on unrelated errors.Feb 28 2022, 10:30 AM
swh-public-ci added a comment.Feb 28 2022, 10:30 AM

Build is green

Patch application report for D7249 (id=26304)

Could not rebase; Attempt merge onto ccff46762b...

Updating ccff4676..74d5567b
Fast-forward
 MANIFEST.in                                        |   1 +
 docs/specs/protocol-reference.rst                  |   4 +-
 docs/specs/swh.xsd                                 |  60 ----
 mypy.ini                                           |   3 +
 requirements-server.txt                            |   1 +
 swh/deposit/api/checks.py                          |  59 +++-
 swh/deposit/tests/api/test_checks.py               | 371 ++++++++++++++++++---
 .../{tests_migration => tests}/test_migrations.py  |   0
 swh/deposit/xsd/codemeta.xsd                       |  16 +
 swh/deposit/xsd/swh.xsd                            |  81 +++++
 10 files changed, 473 insertions(+), 123 deletions(-)
 delete mode 100644 docs/specs/swh.xsd
 rename swh/deposit/{tests_migration => tests}/test_migrations.py (100%)
 create mode 100644 swh/deposit/xsd/codemeta.xsd
 create mode 100644 swh/deposit/xsd/swh.xsd
Changes applied before test
commit 74d5567b31f4e72d37d33b424ff7a8dc57d06b02
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 15:57:40 2022 +0100

    Use xmlschema to validate dates, instead of custom code.
    
    For now this increases code complexity, but this will allow addition
    of other check more easily.

commit b49d5e6ab16bdc53bfcd7b942e4feebac7b19cf8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 13:35:42 2022 +0100

    Enforce create_origin/add_to_origin/reference are mutually exclusive in the schema

commit e087956040e35eee914135b4ec2871b04dd192c4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 13:30:33 2022 +0100

    Add schema validation of <swh:deposit> using swh.xsd

commit 0a6c03965a7a5446884ca07d21b7e03d6ab145c4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 13:24:10 2022 +0100

    Improve readability of pytest errors by removing unnecessary whitespaces
    
    The leading newline prevented textwrap.dedent from removing them.

commit 2e95623c6c7783ed407fa721f09289742ecccab8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 12:13:50 2022 +0100

    Fix swh.xsd
    
    It was never actually tested...

commit 8a7e0832641bda3b4c445793cfb92e3de6c5947e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 24 12:04:03 2022 +0100

    Move tests_migration/test_migrations.py to tests/test_migrations.py
    
    This fixes crashes when running 'pytest -k migration', because swh/deposit/tests_migration/
    lacks a conftest to initialize the database.

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/785/ for more details.

Harbormaster completed remote builds in B27177: Diff 26304.Feb 28 2022, 10:30 AM
Closed by commit rDDEP74d5567b31f4: Use xmlschema to validate dates, instead of custom code. (authored by vlorentz). · Explain WhyFeb 28 2022, 11:01 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDDEP74d5567b31f4: Use xmlschema to validate dates, instead of custom code..

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
deposit/
api/
36 lines100%
Loading...
tests/
api/
209 lines100%
Loading...
xsd/
16 lines--

Diff 26270

View Options

swh/deposit/api/checks.py

Loading...
View Options

swh/deposit/tests/api/test_checks.py

Loading...
View Options

swh/deposit/xsd/codemeta.xsd

Loading...
About · Developer information · Legal