Page MenuHomeSoftware Heritage
Differential D4655

Use msgpack extension type to encode datetime objects in the journal
ClosedPublic
Actions

Authored by douardda on Dec 3 2020, 1:13 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T2834: Use msgpack extension types instead of custom swh encoders/decoders
Commits
rDJNL8dd97125df2d: Use msgpack Timestamp extension type to encode datetime objects in the journal
Summary

The encoding of datetime objects is done using msgpack extension types
(instead of former b'swhtype" custom types).

Support for decoding these later is however provided for backward
compatbility.

Also make the serialization code of the journal independant from
swh.core.api.serializer.

This later move aims at making the msgpack serialization process of the
journal to be specified (and predictable). The code from swh.core is
dedicated at RPC and can thus do much more "custom types" than what is
needed for the journal. So by not using the serialization code from the
swh.core pakage, we make sure we do not inadvertently encode unspecified
objects in the journal.

Related to T2834

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Dec 3 2020, 1:13 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptDec 3 2020, 1:13 PM
swh-public-ci added a comment.Dec 3 2020, 1:15 PM

Build is green

Patch application report for D4655 (id=16509)

Rebasing onto 12b31a2621...

Current branch diff-target is up to date.
Changes applied before test
commit c5122fa000cc5cf164495bb427818548438f7815
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Dec 3 13:05:59 2020 +0100

    Use msgpack extension type to encode datetime objects in the journal
    
    The encoding of datetime objects is done using msgpack extension types
    (instead of former b'swhtype" custom types).
    
    Support for decoding these later is however provided for backward
    compatbility.
    
    Also make the serialization code of the journal independant from
    swh.core.api.serializer.
    
    This later move aims at making the msgpack serialization process of the
    journal to be specified (and predictable). The code from swh.core is
    dedicated at RPC and can thus do much more "custom types" than what is
    needed for the journal. So by not using the serialization code from the
    swh.core pakage, we make sure we do not inadvertently encode unspecified
    objects in the journal.
    
    Related to T2834

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/128/ for more details.

Harbormaster completed remote builds in B17695: Diff 16509.Dec 3 2020, 1:15 PM
douardda mentioned this in D4633: Add a documentation/specification of the journal messages formats.Dec 3 2020, 3:11 PM
olasd added a subscriber: olasd.Dec 3 2020, 3:44 PM

Can we use the built-in msgpack timestamp extension type (code = -1) for timestamps instead: https://msgpack-python.readthedocs.io/en/latest/api.html#msgpack.Timestamp

I think we'll need a conversion function on output (kafka_to_value) but that would be more universal than our own iso8601-based thing.

douardda updated this revision to Diff 16536.Dec 4 2020, 9:51 AM

Use (poorly/not documented) timestapmp/datetime feature flags of mashpack to handle datetime objs

swh-public-ci added a comment.Dec 4 2020, 9:53 AM

Build is green

Patch application report for D4655 (id=16536)

Rebasing onto 12b31a2621...

Current branch diff-target is up to date.
Changes applied before test
commit d3a99afa5271d6c106f32999cfb93ebffd78249e
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Dec 3 13:05:59 2020 +0100

    Use msgpack extension type to encode datetime objects in the journal
    
    The encoding of datetime objects is done using msgpack extension types
    (instead of former b'swhtype" custom types).
    
    Support for decoding these later is however provided for backward
    compatbility.
    
    Also make the serialization code of the journal independant from
    swh.core.api.serializer.
    
    This later move aims at making the msgpack serialization process of the
    journal to be specified (and predictable). The code from swh.core is
    dedicated at RPC and can thus do much more "custom types" than what is
    needed for the journal. So by not using the serialization code from the
    swh.core pakage, we make sure we do not inadvertently encode unspecified
    objects in the journal.
    
    Related to T2834

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/129/ for more details.

Harbormaster completed remote builds in B17723: Diff 16536.Dec 4 2020, 9:53 AM
douardda updated this revision to Diff 16538.Dec 4 2020, 10:06 AM

Simplify a bit the bw compat code (no need for "genericity" there)

swh-public-ci added a comment.Dec 4 2020, 10:08 AM

Build is green

Patch application report for D4655 (id=16538)

Rebasing onto 12b31a2621...

Current branch diff-target is up to date.
Changes applied before test
commit 51b000bb7fe62820ab8f7c8349a005f05c376bb7
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Dec 3 13:05:59 2020 +0100

    Use msgpack extension type to encode datetime objects in the journal
    
    The encoding of datetime objects is done using msgpack extension types
    (instead of former b'swhtype" custom types).
    
    Support for decoding these later is however provided for backward
    compatbility.
    
    Also make the serialization code of the journal independant from
    swh.core.api.serializer.
    
    This later move aims at making the msgpack serialization process of the
    journal to be specified (and predictable). The code from swh.core is
    dedicated at RPC and can thus do much more "custom types" than what is
    needed for the journal. So by not using the serialization code from the
    swh.core pakage, we make sure we do not inadvertently encode unspecified
    objects in the journal.
    
    Related to T2834

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/130/ for more details.

Harbormaster completed remote builds in B17725: Diff 16538.Dec 4 2020, 10:08 AM
douardda updated this revision to Diff 16540.Dec 4 2020, 10:11 AM

Improve (?) the commit message

swh-public-ci added a comment.Dec 4 2020, 10:13 AM

Build is green

Patch application report for D4655 (id=16540)

Rebasing onto 12b31a2621...

Current branch diff-target is up to date.
Changes applied before test
commit 593bd088b0d08ac5b1c1ce339a3a151fcec01cb1
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Dec 3 13:05:59 2020 +0100

    Use msgpack Timestamp extension type to encode datetime objects in the journal
    
    The encoding of datetime objects is done using msgpack extension types
    (instead of former b'swhtype" custom types).
    
    Note that this imply that the timezone of the encoded datime is "lost in
    translation": the resulting datetime will be a (tz-aware) UTC datetime
    object.
    
    Support for decoding these later is however provided for backward
    compatbility.
    
    Also make the serialization code of the journal independant from
    swh.core.api.serializer.
    
    This later move aims at making the msgpack serialization process of the
    journal to be specified (and predictable). The code from swh.core is
    dedicated at RPC and can thus do much more "custom types" than what is
    needed for the journal. So by not using the serialization code from the
    swh.core pakage, we make sure we do not inadvertently encode unspecified
    objects in the journal.
    
    Related to T2834

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/131/ for more details.

Harbormaster completed remote builds in B17726: Diff 16540.Dec 4 2020, 10:13 AM
douardda updated this revision to Diff 16542.Dec 4 2020, 10:45 AM

Add a few tests for msgpack codecs (dates and long integers)

swh-public-ci added a comment.Dec 4 2020, 10:47 AM

Build is green

Patch application report for D4655 (id=16542)

Rebasing onto 12b31a2621...

Current branch diff-target is up to date.
Changes applied before test
commit 8dd97125df2db0eacf73e8b0407837d169bc6b85
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Dec 3 13:05:59 2020 +0100

    Use msgpack Timestamp extension type to encode datetime objects in the journal
    
    The encoding of datetime objects is done using msgpack extension types
    (instead of former b'swhtype" custom types).
    
    Note that this imply that the timezone of the encoded datime is "lost in
    translation": the resulting datetime will be a (tz-aware) UTC datetime
    object.
    
    Support for decoding these later is however provided for backward
    compatbility.
    
    Also make the serialization code of the journal independant from
    swh.core.api.serializer.
    
    This later move aims at making the msgpack serialization process of the
    journal to be specified (and predictable). The code from swh.core is
    dedicated at RPC and can thus do much more "custom types" than what is
    needed for the journal. So by not using the serialization code from the
    swh.core pakage, we make sure we do not inadvertently encode unspecified
    objects in the journal.
    
    Related to T2834

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/132/ for more details.

Harbormaster completed remote builds in B17728: Diff 16542.Dec 4 2020, 10:47 AM
olasd accepted this revision.Dec 4 2020, 2:43 PM
This revision is now accepted and ready to land.Dec 4 2020, 2:43 PM
Closed by commit rDJNL8dd97125df2d: Use msgpack Timestamp extension type to encode datetime objects in the journal (authored by douardda). · Explain WhyDec 4 2020, 2:47 PM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDJNL8dd97125df2d: Use msgpack Timestamp extension type to encode datetime objects in the journal.

Revision Contents
Changeset List

PathSize
swh/
journal/
62 lines
tests/
47 lines

Diff 16560

View Options

swh/journal/serializers.py

Loading...
View Options

swh/journal/tests/test_serializers.py

Loading...
About · Developer information · Legal