Page MenuHomeSoftware Heritage

Add a new ImmutableDict class, and use it in model objects.
ClosedPublic

Authored by vlorentz on Jul 3 2020, 5:08 PM.

Details

Summary

So they are truly immutable now.

Diff Detail

Repository
rDMOD Data model
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Jul 3 2020, 5:08 PM

Build is green

Patch application report for D3417 (id=12114)

Could not rebase; Attempt merge onto 1ff05161e7...

Merge made by the 'recursive' strategy.
 swh/model/collections.py            |  36 ++++
 swh/model/model.py                  | 263 ++++++++++++++++++++++-
 swh/model/tests/test_collections.py |  42 ++++
 swh/model/tests/test_model.py       | 409 ++++++++++++++++++++++++++++++++++++
 4 files changed, 740 insertions(+), 10 deletions(-)
 create mode 100644 swh/model/collections.py
 create mode 100644 swh/model/tests/test_collections.py
Changes applied before test
commit e3fb916401226ce8f46c959894c37ff27a95d76d
Merge: 1ff0516 5b7ac4f
Author: Jenkins user <jenkins@localhost>
Date:   Fri Jul 3 15:08:49 2020 +0000

    Merge branch 'diff-target' into HEAD

commit 5b7ac4fdf7c465def8303f2184ae9e0097983eca
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit 5e3ae76689895f5bccb62cccca813f3810231a85
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/101/ for more details.

ardumont added inline comments.
swh/model/collections.py
11

What does KT and VT stands for?

KeyType and ValueType?

Seems right ;)

olasd added a subscriber: olasd.Jul 4 2020, 1:54 PM

That's great, thanks!

Does the serialization/deserialization of these data structures work ok? I think we'll need to do a "deep" (well, 2D really) tuple conversion, instead of just top level tuplification, as msgpack turns all tuples into lists.

I think ImmutableDict[str, object] should really be spelled ImmutableDict[str, Any]. Both versions are used currently.

In D3417#84040, @olasd wrote:

Does the serialization/deserialization of these data structures work ok? I think we'll need to do a "deep" (well, 2D really) tuple conversion, instead of just top level tuplification, as msgpack turns all tuples into lists.

Probably not. I assumed we had tests for it, but it looks like we don't.

swh/model/collections.py
11

Yes. I'm reusing the names from typing.py.

vlorentz updated this revision to Diff 12134.Jul 6 2020, 11:59 AM

fix base commit

Build is green

Patch application report for D3417 (id=12132)

Rebasing onto 1ff05161e7...

Current branch diff-target is up to date.
Changes applied before test
commit 30e0967fbdb38701eb9f4fa16f31a5c0b74b4a79
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit da01882e0d68bfd7797a8e4e7502ab90906358ee
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/104/ for more details.

Build is green

Patch application report for D3417 (id=12134)

Could not rebase; Attempt merge onto 1ff05161e7...

Updating 1ff0516..408e638
Fast-forward
 swh/model/collections.py            |  36 ++++
 swh/model/model.py                  | 255 +++++++++++++++++++++-
 swh/model/tests/test_collections.py |  42 ++++
 swh/model/tests/test_model.py       | 407 ++++++++++++++++++++++++++++++++++++
 4 files changed, 730 insertions(+), 10 deletions(-)
 create mode 100644 swh/model/collections.py
 create mode 100644 swh/model/tests/test_collections.py
Changes applied before test
commit 408e638b1dffe99e1afb1a3c77dbe0e14cb56042
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit d4ea9a56aca02b8aed15186372749bfbe639d69f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/106/ for more details.

In D3417#84040, @olasd wrote:

Does the serialization/deserialization of these data structures work ok? I think we'll need to do a "deep" (well, 2D really) tuple conversion, instead of just top level tuplification, as msgpack turns all tuples into lists.

Probably not. I assumed we had tests for it, but it looks like we don't.

Actually, yes it does, because dictify converts ImmutableDict to dict

ardumont accepted this revision.Mon, Jul 6, 5:18 PM
This revision is now accepted and ready to land.Mon, Jul 6, 5:18 PM

Build has FAILED

Patch application report for D3417 (id=12184)

Could not rebase; Attempt merge onto a7d9aca2b0...

Updating a7d9aca..7e870da
Fast-forward
 swh/model/collections.py            |  36 ++++
 swh/model/model.py                  | 255 +++++++++++++++++++++-
 swh/model/tests/test_collections.py |  42 ++++
 swh/model/tests/test_model.py       | 407 ++++++++++++++++++++++++++++++++++++
 4 files changed, 730 insertions(+), 10 deletions(-)
 create mode 100644 swh/model/collections.py
 create mode 100644 swh/model/tests/test_collections.py
Changes applied before test
commit 7e870da44f87655ce027876a3b3885d60b8ffc57
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit 78fc5f7cbc500bde5d917b8f55fc6c9e79608479
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

Link to build: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/110/
See console output for more information: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/110/console

vlorentz updated this revision to Diff 12198.Tue, Jul 7, 1:20 PM

rebase + update for extra_headers / copy_pop

Build has FAILED

Patch application report for D3417 (id=12198)

Rebasing onto a7d9aca2b0...

Current branch diff-target is up to date.
Changes applied before test
commit 2628ba1ab3e2163a043c2f1571c7a2a640d12cb0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit 78fc5f7cbc500bde5d917b8f55fc6c9e79608479
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

Link to build: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/112/
See console output for more information: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/112/console

vlorentz updated this revision to Diff 12201.Tue, Jul 7, 1:31 PM

rebase + add test + fix type annotation

Build is green

Patch application report for D3417 (id=12201)

Could not rebase; Attempt merge onto a7d9aca2b0...

Updating a7d9aca..af0dd1a
Fast-forward
 swh/model/collections.py            |  49 +++++
 swh/model/model.py                  | 263 +++++++++++++++++++++--
 swh/model/tests/test_collections.py |  50 +++++
 swh/model/tests/test_model.py       | 407 ++++++++++++++++++++++++++++++++++++
 4 files changed, 754 insertions(+), 15 deletions(-)
 create mode 100644 swh/model/collections.py
 create mode 100644 swh/model/tests/test_collections.py
Changes applied before test
commit af0dd1aaee4943d6cec0f50f77a14bcd4d8733b5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit 78fc5f7cbc500bde5d917b8f55fc6c9e79608479
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/115/ for more details.

ardumont accepted this revision.Tue, Jul 7, 1:40 PM