Page MenuHomeSoftware Heritage
Differential D3007

Add a key pretty printer to the serializers module
ClosedPublic
Actions

Authored by olasd on Apr 10 2020, 1:19 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDJNLa20e557a824c: Add a key pretty printer to the serializers module
Summary

Depends on D3006.

Test Plan

new tox test added

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

olasd created this revision.Apr 10 2020, 1:19 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 10 2020, 1:19 PM
swh-public-ci added a comment.Apr 10 2020, 1:23 PM

Build is green

Patch application report for D3007 (id=10669)

Could not rebase; Attempt merge onto 750d4c6c5b...

Updating 750d4c6..6ef3ee8
Fast-forward
 swh/journal/serializers.py             | 88 +++++++++++++++++++++++++++++++++-
 swh/journal/tests/conftest.py          | 41 ++++++++++++----
 swh/journal/tests/test_kafka_writer.py | 76 ++++++++++++-----------------
 swh/journal/tests/test_replay.py       | 38 +++++++--------
 swh/journal/tests/test_serializers.py  | 26 ++++++++++
 swh/journal/writer/kafka.py            | 60 +++++------------------
 6 files changed, 204 insertions(+), 125 deletions(-)
Changes applied before test
commit 6ef3ee811cd6206f3ac50e2ef8d7cf69f087fb97
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 9 18:00:10 2020 +0200

    Add a key pretty printer to the serializers module

commit b7a6b52867ffb38703bdcd1082bb4ca6899e3159
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 10 12:45:59 2020 +0200

    Ensure all object keys are passed through unharmed by the journal
    
    The previous implementation of this test only checked keys that were plain
    bytes. It turns out that the "compounded" keys decoding was wrong.
    
    This allows us to simplify the TEST_OBJECT_DICTS structure to only contain a
    list of object dicts.

commit fb21467e8864d2e6e71601038ef0a160b1a33a4f
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 9 17:58:40 2020 +0200

    Move _get_key from the kafka writer to serializers
    
    Use the opportunity to add some trivial smoke tests

commit 9f0cd745d07e7e379446eb82bcfd25e88e6d1d4a
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 10 11:41:54 2020 +0200

    Add a model object based version of the journal writer test objects
    
    This refactors the various conversions happening in the journal writer tests in
    a single place.
    
    When developing this, I noticed that the journal writer would let Content data
    go through; Make sure we do not let it do that.

commit 1c9ccb29eea3b974ea0d3613d61a65cd4e83037b
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 9 18:50:18 2020 +0200

    Rename OBJECT_TYPE_KEYS to TEST_OBJECT_DICTS

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/37/ for more details.

Harbormaster completed remote builds in B11787: Diff 10669.Apr 10 2020, 1:23 PM
olasd added a child revision: D2994: Add delivery notification handling to swh.journal.writer.kafka.Apr 10 2020, 1:29 PM
olasd updated this revision to Diff 10671.Apr 10 2020, 1:31 PM
olasd added a subscriber: vlorentz.

Apply @vlorentz's comment from D2994

swh-public-ci added a comment.Apr 10 2020, 1:39 PM

Build is green

Patch application report for D3007 (id=10671)

Could not rebase; Attempt merge onto 750d4c6c5b...

Updating 750d4c6..5df65c5
Fast-forward
 swh/journal/serializers.py             | 88 +++++++++++++++++++++++++++++++++-
 swh/journal/tests/conftest.py          | 41 ++++++++++++----
 swh/journal/tests/test_kafka_writer.py | 76 ++++++++++++-----------------
 swh/journal/tests/test_replay.py       | 38 +++++++--------
 swh/journal/tests/test_serializers.py  | 26 ++++++++++
 swh/journal/writer/kafka.py            | 60 +++++------------------
 6 files changed, 204 insertions(+), 125 deletions(-)
Changes applied before test
commit 5df65c50777325ae710f51a0e93ee94af2791f43
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 9 18:00:10 2020 +0200

    Add a key pretty printer to the serializers module

commit b7a6b52867ffb38703bdcd1082bb4ca6899e3159
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 10 12:45:59 2020 +0200

    Ensure all object keys are passed through unharmed by the journal
    
    The previous implementation of this test only checked keys that were plain
    bytes. It turns out that the "compounded" keys decoding was wrong.
    
    This allows us to simplify the TEST_OBJECT_DICTS structure to only contain a
    list of object dicts.

commit fb21467e8864d2e6e71601038ef0a160b1a33a4f
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 9 17:58:40 2020 +0200

    Move _get_key from the kafka writer to serializers
    
    Use the opportunity to add some trivial smoke tests

commit 9f0cd745d07e7e379446eb82bcfd25e88e6d1d4a
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 10 11:41:54 2020 +0200

    Add a model object based version of the journal writer test objects
    
    This refactors the various conversions happening in the journal writer tests in
    a single place.
    
    When developing this, I noticed that the journal writer would let Content data
    go through; Make sure we do not let it do that.

commit 1c9ccb29eea3b974ea0d3613d61a65cd4e83037b
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 9 18:50:18 2020 +0200

    Rename OBJECT_TYPE_KEYS to TEST_OBJECT_DICTS

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/39/ for more details.

Harbormaster completed remote builds in B11789: Diff 10671.Apr 10 2020, 1:39 PM
olasd updated this revision to Diff 10688.Apr 10 2020, 4:32 PM

Rebase on top of updated D3006

vlorentz added a comment.Apr 10 2020, 4:40 PM

didn't I already see this code somewhere else?

swh-public-ci added a comment.Apr 10 2020, 4:44 PM

Build is green

Patch application report for D3007 (id=10688)

Could not rebase; Attempt merge onto 9f0cd745d0...

Updating 9f0cd74..a20e557
Fast-forward
 swh/journal/serializers.py             | 80 +++++++++++++++++++++++++++++++++-
 swh/journal/tests/conftest.py          | 20 ++++-----
 swh/journal/tests/test_kafka_writer.py | 23 +++++-----
 swh/journal/tests/test_replay.py       | 40 ++++++++---------
 swh/journal/tests/test_serializers.py  | 26 +++++++++++
 swh/journal/writer/kafka.py            | 58 ++++--------------------
 6 files changed, 153 insertions(+), 94 deletions(-)
Changes applied before test
commit a20e557a824c35ee4ebdb03b28504be138ba0baf
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 9 18:00:10 2020 +0200

    Add a key pretty printer to the serializers module

commit 63bd07af498b4d65d35ea613a1815dc2b51d0a4d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 10 12:45:59 2020 +0200

    Ensure all object keys are passed through unharmed by the journal
    
    The previous implementation of this test only checked keys that were plain
    bytes. It turns out that the "compounded" keys decoding was wrong.
    
    This allows us to simplify the TEST_OBJECT_DICTS structure to only contain a
    list of object dicts.

commit cc5797df4b08c2493155997fcde85e9155e18e31
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 9 17:58:40 2020 +0200

    Move _get_key from the kafka writer to serializers
    
    Use the opportunity to add some trivial smoke tests

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/43/ for more details.

Harbormaster completed remote builds in B11809: Diff 10688.Apr 10 2020, 4:44 PM
olasd added a comment.Apr 10 2020, 5:03 PM
In D3007#72823, @vlorentz wrote:

didn't I already see this code somewhere else?

Yeah, I split it out of D2994.

vlorentz accepted this revision.Apr 14 2020, 10:48 AM
This revision is now accepted and ready to land.Apr 14 2020, 10:48 AM
Closed by commit rDJNLa20e557a824c: Add a key pretty printer to the serializers module (authored by olasd). · Explain WhyApr 14 2020, 12:10 PM
This revision was automatically updated to reflect the committed changes.
olasd added a commit: rDJNLa20e557a824c: Add a key pretty printer to the serializers module.

Revision Contents
Changeset List

PathSize
swh/
journal/
22 lines
tests/
17 lines

Diff 10713

View Options

swh/journal/serializers.py

Loading...
View Options

swh/journal/tests/test_serializers.py

Loading...
About · Developer information · Legal