Page MenuHomeSoftware Heritage

Add raw metadata to the model.
ClosedPublic

Authored by vlorentz on Jul 2 2020, 5:53 PM.

Details

Summary

This will allow swh-storage to have a signature for *_metadata_add
that is consistent with other *_add endpoints.

Diff Detail

Repository
rDMOD Data model
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Jul 2 2020, 5:53 PM

Build is green

Patch application report for D3396 (id=12046)

Rebasing onto 8863b5c186...

Current branch diff-target is up to date.
Changes applied before test
commit 31a9e332ebb7f2bb7dfb79e42418a05e48e27a02
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/94/ for more details.

Looks promising ;)

douardda requested changes to this revision.Jul 3 2020, 11:32 AM
douardda added a subscriber: douardda.

Any reason not to use type_validator() in these new schema entities?

swh/model/model.py
715

can't we directly take the path of immutability while at it here, e.g. using a Tuple[Tuple[bytes, bytes], ...] as metadata type instead?

748

why not an Optional[int] here? what this visit refers to? the visit of the metadata fetcher tool? If so it should better be moved in the "# source" block below. If not, it must refers to the origin visit (by a loader), in which case I do not really see the need for this field.

This revision now requires changes to proceed.Jul 3 2020, 11:32 AM
vlorentz added inline comments.Jul 3 2020, 11:33 AM
swh/model/model.py
715

ok

748

typo

vlorentz updated this revision to Diff 12080.Jul 3 2020, 11:45 AM
  • apply comments
  • add defaults
  • add more type validation
  • add basic validation tests for 'id'

Build is green

Patch application report for D3396 (id=12080)

Rebasing onto 8863b5c186...

Current branch diff-target is up to date.
Changes applied before test
commit 12f8b52c992b31ceb843cae5e0d028cacc5a842a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/95/ for more details.

vlorentz marked an inline comment as done.Jul 3 2020, 11:48 AM
vlorentz updated this revision to Diff 12082.Jul 3 2020, 11:57 AM

Add tests for origin validation.

vlorentz planned changes to this revision.Jul 3 2020, 11:57 AM

Build is green

Patch application report for D3396 (id=12082)

Rebasing onto 8863b5c186...

Current branch diff-target is up to date.
Changes applied before test
commit 4a4d461c391a22c4f99c9504c100d6b1d193de56
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/96/ for more details.

vlorentz updated this revision to Diff 12084.Jul 3 2020, 12:12 PM

Add tests for visit and snapshot validation.

vlorentz planned changes to this revision.Jul 3 2020, 12:12 PM

Build is green

Patch application report for D3396 (id=12084)

Rebasing onto 8863b5c186...

Current branch diff-target is up to date.
Changes applied before test
commit d4953992a857cffd0fdabc7ff41ab74d6c7736ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/98/ for more details.

vlorentz updated this revision to Diff 12107.Jul 3 2020, 3:51 PM

Add tests for release/revision/directory/path validation.

Build is green

Patch application report for D3396 (id=12107)

Rebasing onto 8863b5c186...

Current branch diff-target is up to date.
Changes applied before test
commit 5e3ae76689895f5bccb62cccca813f3810231a85
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/100/ for more details.

heads up on D3412 btw.

heads up on D3412 btw.

Sorry to insist on D3412 but it landed and it's now tagged in model.

So we'll be flooded with deprecation notice if you don't change it (tests, prod also).

The gist of the new naming convention is to rename accordingly:

  • PersistentId: SWHID
  • persistent_identifier: swhid
  • parse_persistent_identifier: parse_swhid

Can you please adapt?

(I also see that your next diff building on this one keep the old names)

Thanks in advance,

vlorentz updated this revision to Diff 12131.Jul 6 2020, 11:57 AM

rename to SWHID.

Build is green

Patch application report for D3396 (id=12131)

Rebasing onto 1ff05161e7...

Current branch diff-target is up to date.
Changes applied before test
commit d4ea9a56aca02b8aed15186372749bfbe639d69f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/103/ for more details.

ardumont accepted this revision.Mon, Jul 6, 5:59 PM

Looks good

(thx about the renaming ;)

douardda accepted this revision.Tue, Jul 7, 11:26 AM

As said on IRC, I'd rather see this diff depends on D3417, but otherwise I'm fine with it.

This revision is now accepted and ready to land.Tue, Jul 7, 11:26 AM

Build is green

Patch application report for D3396 (id=12183)

Rebasing onto a7d9aca2b0...

Current branch diff-target is up to date.
Changes applied before test
commit 78fc5f7cbc500bde5d917b8f55fc6c9e79608479
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/109/ for more details.

Build is green

Patch application report for D3396 (id=12200)

Rebasing onto a7d9aca2b0...

Current branch diff-target is up to date.
Changes applied before test
commit 78fc5f7cbc500bde5d917b8f55fc6c9e79608479
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/114/ for more details.

This revision was automatically updated to reflect the committed changes.