Details

Reviewers

Group Reviewers

Commits

rDSTOC3a6c5e354684: Make content_add validate its input using swh-model.
rDSTO3a6c5e354684: Make content_add validate its input using swh-model.

Summary

The initial goal of this commit was to make the in-mem content_add raise
errors on input that would make postgresql error.

As a side-effect, the pgsql content_add now validate its input as well,
so clients get a nice/meaningful error instead of postgresql's error.

I'm planning on adding similar validation to other *_add methods.

Depends on D1864 (same repo), and D1860, D1861, D1862, D1863 (swh-model)

Diff Detail

Repository

rDSTO Storage manager

Branch

content_add-validation

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 7468
Build 10660: tox-on-jenkins	Jenkins
Build 10659: arc lint + arc unit

Event Timeline

vlorentz created this revision.Aug 19 2019, 2:45 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptAug 19 2019, 2:45 PM

vlorentz edited the summary of this revision. (Show Details)Aug 19 2019, 2:46 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/576/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/576/console

Harbormaster failed remote builds in B7297: Diff 6261!Aug 19 2019, 2:52 PM

vlorentz retitled this revision from Make content_add validate its input. to Make content_add validate its input using swh-model..Aug 19 2019, 2:52 PM

vlorentz edited the summary of this revision. (Show Details)Aug 19 2019, 3:06 PM

Use from_dict instead of constructor.

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/578/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/578/console

Harbormaster failed remote builds in B7298: Diff 6262!Aug 19 2019, 3:18 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/580/ for more details.

Harbormaster completed remote builds in B7298: Diff 6262.Aug 20 2019, 12:23 PM

really use from_dict instead of constructor
update required swh-model version.

Build was aborted

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/581/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/581/console

Harbormaster failed remote builds in B7331: Diff 6294!Aug 20 2019, 12:25 PM

remove useless deepcopies

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/582/ for more details.

Harbormaster completed remote builds in B7332: Diff 6295.Aug 20 2019, 12:32 PM

rebase

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/587/ for more details.

Harbormaster completed remote builds in B7340: Diff 6303.Aug 20 2019, 3:04 PM

Make postgresql validate itself when possible. (Depends on D1872 for the tests.)

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/588/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/588/console

Harbormaster failed remote builds in B7342: Diff 6305!Aug 20 2019, 4:39 PM

vlorentz added a child revision: D1873: Make directory_add validate its input..Aug 20 2019, 5:20 PM

vlorentz added a parent revision: D1872: Make copy_to raise exceptions from the writer thread..Aug 20 2019, 5:26 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/596/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/596/console

vlorentz added a parent revision: D1889: Use explicit arguments for origin_visit_add..Aug 22 2019, 2:20 PM

rebase

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/600/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/600/console

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/601/ for more details.

Harbormaster completed remote builds in B7381: Diff 6344.Aug 22 2019, 2:42 PM

vlorentz removed parent revisions: D1872: Make copy_to raise exceptions from the writer thread., D1864: Remove test_content_missing__marked_missing, it makes no sense..Aug 22 2019, 3:36 PM

rebase

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/608/ for more details.

Harbormaster completed remote builds in B7387: Diff 6350.Aug 22 2019, 3:47 PM

vlorentz removed a parent revision: D1864: Remove test_content_missing__marked_missing, it makes no sense..Aug 22 2019, 3:50 PM

vlorentz removed a parent revision: D1889: Use explicit arguments for origin_visit_add..Aug 22 2019, 3:53 PM

Looks good.

But _filter_new_content is no longer a filter.
It normalizes and validates the content dict.
So that needs a rename.

Also the internal methods need docstrings to add details on what's what (and also to make sure we do not diverge in the future for any unforeseen reasons).

Cheers,

This revision now requires changes to proceed.Aug 26 2019, 4:23 PM