Page MenuHomeSoftware Heritage

Add raw metadata to the model.
ClosedPublic

Authored by vlorentz on Jul 2 2020, 5:53 PM.

Details

Summary

This will allow swh-storage to have a signature for *_metadata_add
that is consistent with other *_add endpoints.

Diff Detail

Repository
rDMOD Data model
Branch
metadata-model
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 13301
Build 20323: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 20322: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D3396 (id=12046)

Rebasing onto 8863b5c186...

Current branch diff-target is up to date.
Changes applied before test
commit 31a9e332ebb7f2bb7dfb79e42418a05e48e27a02
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/94/ for more details.

douardda added a subscriber: douardda.

Any reason not to use type_validator() in these new schema entities?

swh/model/model.py
688

can't we directly take the path of immutability while at it here, e.g. using a Tuple[Tuple[bytes, bytes], ...] as metadata type instead?

721

why not an Optional[int] here? what this visit refers to? the visit of the metadata fetcher tool? If so it should better be moved in the "# source" block below. If not, it must refers to the origin visit (by a loader), in which case I do not really see the need for this field.

This revision now requires changes to proceed.Jul 3 2020, 11:32 AM
swh/model/model.py
688

ok

721

typo

  • apply comments
  • add defaults
  • add more type validation
  • add basic validation tests for 'id'

Build is green

Patch application report for D3396 (id=12080)

Rebasing onto 8863b5c186...

Current branch diff-target is up to date.
Changes applied before test
commit 12f8b52c992b31ceb843cae5e0d028cacc5a842a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/95/ for more details.

Add tests for origin validation.

Build is green

Patch application report for D3396 (id=12082)

Rebasing onto 8863b5c186...

Current branch diff-target is up to date.
Changes applied before test
commit 4a4d461c391a22c4f99c9504c100d6b1d193de56
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/96/ for more details.

Add tests for visit and snapshot validation.

Build is green

Patch application report for D3396 (id=12084)

Rebasing onto 8863b5c186...

Current branch diff-target is up to date.
Changes applied before test
commit d4953992a857cffd0fdabc7ff41ab74d6c7736ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/98/ for more details.

Add tests for release/revision/directory/path validation.

Build is green

Patch application report for D3396 (id=12107)

Rebasing onto 8863b5c186...

Current branch diff-target is up to date.
Changes applied before test
commit 5e3ae76689895f5bccb62cccca813f3810231a85
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/100/ for more details.

heads up on D3412 btw.

Sorry to insist on D3412 but it landed and it's now tagged in model.

So we'll be flooded with deprecation notice if you don't change it (tests, prod also).

The gist of the new naming convention is to rename accordingly:

  • PersistentId: SWHID
  • persistent_identifier: swhid
  • parse_persistent_identifier: parse_swhid

Can you please adapt?

(I also see that your next diff building on this one keep the old names)

Thanks in advance,

Build is green

Patch application report for D3396 (id=12131)

Rebasing onto 1ff05161e7...

Current branch diff-target is up to date.
Changes applied before test
commit d4ea9a56aca02b8aed15186372749bfbe639d69f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/103/ for more details.

Looks good

(thx about the renaming ;)

As said on IRC, I'd rather see this diff depends on D3417, but otherwise I'm fine with it.

This revision is now accepted and ready to land.Jul 7 2020, 11:26 AM

Build is green

Patch application report for D3396 (id=12183)

Rebasing onto a7d9aca2b0...

Current branch diff-target is up to date.
Changes applied before test
commit 78fc5f7cbc500bde5d917b8f55fc6c9e79608479
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/109/ for more details.

Build is green

Patch application report for D3396 (id=12200)

Rebasing onto a7d9aca2b0...

Current branch diff-target is up to date.
Changes applied before test
commit 78fc5f7cbc500bde5d917b8f55fc6c9e79608479
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/114/ for more details.

This revision was automatically updated to reflect the committed changes.