api.checks: Warn when suggested fields are missing from metadata
ClosedPublic
Actions

Authored by ardumont on Feb 21 2022, 4:00 PM.

Details

Reviewers

moranegg
vlorentz

Group Reviewers

Reviewers

Maniphest Tasks

T3677: Separate origin-source-code and provenance-metadata in the deposit

Commits

rDDEP339f7dd390ca: api.checks: Warn when suggested fields are missing from metadata

Summary

This introduces a new check about the metadata provenance. While it's a suggested field,
it's definitely something that we want deposit clients to send us. So warn when it's not
the case. That does not reject the deposit but it's worth keeping that detail in the
backend.

Related to T3677
Depends on D7200

Test Plan

tox

Diff Detail

Repository

rDDEP Push deposit

Branch

master

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 27002
Build 42218: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 42217: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	210 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.deposit.tests.api.test_deposit_private_check::test_deposit_ko_unsupported_tarball tmp_path = '/tmp/pytest-of-jenkins/pytest-0/test_deposit_ko_unsupported_ta0' authenticated_client = <rest_framework.test.APIClient object at 0x7f6e3db08358> deposit_collection = <DepositCollection: {'id': 104, 'name': 'test'}>
	87 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.deposit.tests.cli.test_client::test_cli_client_generate_metadata_ok slug = '9e61f8c3-3c28-411d-a975-4cee69860c4e' def test_cli_client_generate_metadata_ok(slug):
	94 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.deposit.tests.cli.test_client::test_cli_client_generate_metadata_ok2 slug = '54a240db-801c-43d1-93d3-afbdb135d2c7' def test_cli_client_generate_metadata_ok2(slug):
	555 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.deposit.tests.api.test_basic_auth::test_service_document_basic
	104 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.deposit.tests.api.test_checks::test_api_checks_check_metadata_fields_ko_and_missing_suggested_fields[metadata_ko0-expected_invalid_summary0]
		View Full Test Results (3 Failed · 286 Passed)

Event Timeline

ardumont created this revision.Feb 21 2022, 4:00 PM

ardumont added a child revision: D7210: deposit_check: Actually store warning in deposit status detail.

ardumont edited the summary of this revision. (Show Details)Feb 21 2022, 4:02 PM

ardumont added a parent revision: D7200: deposit: Add utils to parse metadata provenance from atom entry file.

Build has FAILED

Patch application report for D7209 (id=26131)

Rebasing onto 40adc8c23b...

Current branch diff-target is up to date.

Changes applied before test

commit b9749edc15f90a63c12f47e022a92eecf27161e6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Feb 21 15:26:13 2022 +0100

    api.checks: Warn when suggested fields are missing from metadata
    
    This introduces a new check about the metadata provenance. While it's a suggested field,
    it's definitely something that we want deposit clients to send us. So warn when it's not
    the case. That does not reject the deposit but it's worth keeping that detail in the
    backend.
    
    Related to T3677

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/707/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/707/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 21 2022, 4:03 PM

Harbormaster failed remote builds in B27002: Diff 26131!

Rework commit and fix tests (i had forgotten to run all tests on that code ¯\_(ツ)_/¯)

Build is green

Patch application report for D7209 (id=26136)

Rebasing onto 40adc8c23b...

Current branch diff-target is up to date.

Changes applied before test

commit 339f7dd390ca8cbe8e79a77a1bc2e3c704d5f3f1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Feb 21 15:26:13 2022 +0100

    api.checks: Warn when suggested fields are missing from metadata
    
    This introduces a new check about the metadata provenance. While it's a suggested field,
    it's definitely something that we want deposit clients to send us. So warn when it's not
    the case. That does not reject the deposit but it's worth keeping that detail in the
    backend.
    
    Related to T3677

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/709/ for more details.

Harbormaster completed remote builds in B27007: Diff 26136.Feb 21 2022, 4:34 PM

ardumont requested review of this revision.Feb 21 2022, 4:34 PM

vlorentz accepted this revision.Feb 21 2022, 6:07 PM

This revision is now accepted and ready to land.Feb 21 2022, 6:07 PM

Closed by commit rDDEP339f7dd390ca: api.checks: Warn when suggested fields are missing from metadata (authored by ardumont). · Explain WhyFeb 21 2022, 6:23 PM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDDEP339f7dd390ca: api.checks: Warn when suggested fields are missing from metadata.

ardumont mentioned this in T3935: Change metadata-only deposits protocol to add a metadata-provenance url.Feb 23 2022, 6:52 PM