Page MenuHomeSoftware Heritage

Add a new ImmutableDict class, and use it in model objects.
ClosedPublic

Authored by vlorentz on Jul 3 2020, 5:08 PM.

Details

Summary

So they are truly immutable now.

Diff Detail

Event Timeline

Build is green

Patch application report for D3417 (id=12114)

Could not rebase; Attempt merge onto 1ff05161e7...

Merge made by the 'recursive' strategy.
 swh/model/collections.py            |  36 ++++
 swh/model/model.py                  | 263 ++++++++++++++++++++++-
 swh/model/tests/test_collections.py |  42 ++++
 swh/model/tests/test_model.py       | 409 ++++++++++++++++++++++++++++++++++++
 4 files changed, 740 insertions(+), 10 deletions(-)
 create mode 100644 swh/model/collections.py
 create mode 100644 swh/model/tests/test_collections.py
Changes applied before test
commit e3fb916401226ce8f46c959894c37ff27a95d76d
Merge: 1ff0516 5b7ac4f
Author: Jenkins user <jenkins@localhost>
Date:   Fri Jul 3 15:08:49 2020 +0000

    Merge branch 'diff-target' into HEAD

commit 5b7ac4fdf7c465def8303f2184ae9e0097983eca
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit 5e3ae76689895f5bccb62cccca813f3810231a85
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/101/ for more details.

ardumont added inline comments.
swh/model/collections.py
10

What does KT and VT stands for?

KeyType and ValueType?

That's great, thanks!

Does the serialization/deserialization of these data structures work ok? I think we'll need to do a "deep" (well, 2D really) tuple conversion, instead of just top level tuplification, as msgpack turns all tuples into lists.

I think ImmutableDict[str, object] should really be spelled ImmutableDict[str, Any]. Both versions are used currently.

In D3417#84040, @olasd wrote:

Does the serialization/deserialization of these data structures work ok? I think we'll need to do a "deep" (well, 2D really) tuple conversion, instead of just top level tuplification, as msgpack turns all tuples into lists.

Probably not. I assumed we had tests for it, but it looks like we don't.

swh/model/collections.py
10

Yes. I'm reusing the names from typing.py.

Build is green

Patch application report for D3417 (id=12132)

Rebasing onto 1ff05161e7...

Current branch diff-target is up to date.
Changes applied before test
commit 30e0967fbdb38701eb9f4fa16f31a5c0b74b4a79
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit da01882e0d68bfd7797a8e4e7502ab90906358ee
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/104/ for more details.

Build is green

Patch application report for D3417 (id=12134)

Could not rebase; Attempt merge onto 1ff05161e7...

Updating 1ff0516..408e638
Fast-forward
 swh/model/collections.py            |  36 ++++
 swh/model/model.py                  | 255 +++++++++++++++++++++-
 swh/model/tests/test_collections.py |  42 ++++
 swh/model/tests/test_model.py       | 407 ++++++++++++++++++++++++++++++++++++
 4 files changed, 730 insertions(+), 10 deletions(-)
 create mode 100644 swh/model/collections.py
 create mode 100644 swh/model/tests/test_collections.py
Changes applied before test
commit 408e638b1dffe99e1afb1a3c77dbe0e14cb56042
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit d4ea9a56aca02b8aed15186372749bfbe639d69f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/106/ for more details.

In D3417#84040, @olasd wrote:

Does the serialization/deserialization of these data structures work ok? I think we'll need to do a "deep" (well, 2D really) tuple conversion, instead of just top level tuplification, as msgpack turns all tuples into lists.

Probably not. I assumed we had tests for it, but it looks like we don't.

Actually, yes it does, because dictify converts ImmutableDict to dict

This revision is now accepted and ready to land.Jul 6 2020, 5:18 PM

Build has FAILED

Patch application report for D3417 (id=12184)

Could not rebase; Attempt merge onto a7d9aca2b0...

Updating a7d9aca..7e870da
Fast-forward
 swh/model/collections.py            |  36 ++++
 swh/model/model.py                  | 255 +++++++++++++++++++++-
 swh/model/tests/test_collections.py |  42 ++++
 swh/model/tests/test_model.py       | 407 ++++++++++++++++++++++++++++++++++++
 4 files changed, 730 insertions(+), 10 deletions(-)
 create mode 100644 swh/model/collections.py
 create mode 100644 swh/model/tests/test_collections.py
Changes applied before test
commit 7e870da44f87655ce027876a3b3885d60b8ffc57
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit 78fc5f7cbc500bde5d917b8f55fc6c9e79608479
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

Link to build: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/110/
See console output for more information: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/110/console

rebase + update for extra_headers / copy_pop

Build has FAILED

Patch application report for D3417 (id=12198)

Rebasing onto a7d9aca2b0...

Current branch diff-target is up to date.
Changes applied before test
commit 2628ba1ab3e2163a043c2f1571c7a2a640d12cb0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit 78fc5f7cbc500bde5d917b8f55fc6c9e79608479
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

Link to build: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/112/
See console output for more information: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/112/console

rebase + add test + fix type annotation

Build is green

Patch application report for D3417 (id=12201)

Could not rebase; Attempt merge onto a7d9aca2b0...

Updating a7d9aca..af0dd1a
Fast-forward
 swh/model/collections.py            |  49 +++++
 swh/model/model.py                  | 263 +++++++++++++++++++++--
 swh/model/tests/test_collections.py |  50 +++++
 swh/model/tests/test_model.py       | 407 ++++++++++++++++++++++++++++++++++++
 4 files changed, 754 insertions(+), 15 deletions(-)
 create mode 100644 swh/model/collections.py
 create mode 100644 swh/model/tests/test_collections.py
Changes applied before test
commit af0dd1aaee4943d6cec0f50f77a14bcd4d8733b5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 3 17:08:19 2020 +0200

    Add a new ImmutableDict class, and use it in model objects.
    
    So they are truly immutable now.

commit 78fc5f7cbc500bde5d917b8f55fc6c9e79608479
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jul 2 17:53:13 2020 +0200

    Add raw metadata to the model.
    
    This will allow swh-storage to have a signature for *_metadata_add
    that is consistent with other *_add endpoints.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/115/ for more details.