Page MenuHomeSoftware Heritage

Ensure all object keys are passed through unharmed by the journal
ClosedPublic

Authored by olasd on Apr 10 2020, 1:16 PM.

Details

Summary

The previous implementation of this test only checked keys that were plain
bytes. It turns out that the "compounded" keys decoding was wrong.

This allows us to simplify the TEST_OBJECT_DICTS structure to only contain a
list of object dicts.

Depends on D3005.

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D3006 (id=10668)

Could not rebase; Attempt merge onto 750d4c6c5b...

Updating 750d4c6..b7a6b52
Fast-forward
 swh/journal/serializers.py             | 66 ++++++++++++++++++++++++++++-
 swh/journal/tests/conftest.py          | 41 +++++++++++++-----
 swh/journal/tests/test_kafka_writer.py | 76 ++++++++++++++--------------------
 swh/journal/tests/test_replay.py       | 38 ++++++++---------
 swh/journal/tests/test_serializers.py  |  9 ++++
 swh/journal/writer/kafka.py            | 60 +++++----------------------
 6 files changed, 165 insertions(+), 125 deletions(-)
Changes applied before test
commit b7a6b52867ffb38703bdcd1082bb4ca6899e3159
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 10 12:45:59 2020 +0200

    Ensure all object keys are passed through unharmed by the journal
    
    The previous implementation of this test only checked keys that were plain
    bytes. It turns out that the "compounded" keys decoding was wrong.
    
    This allows us to simplify the TEST_OBJECT_DICTS structure to only contain a
    list of object dicts.

commit fb21467e8864d2e6e71601038ef0a160b1a33a4f
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 9 17:58:40 2020 +0200

    Move _get_key from the kafka writer to serializers
    
    Use the opportunity to add some trivial smoke tests

commit 9f0cd745d07e7e379446eb82bcfd25e88e6d1d4a
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 10 11:41:54 2020 +0200

    Add a model object based version of the journal writer test objects
    
    This refactors the various conversions happening in the journal writer tests in
    a single place.
    
    When developing this, I noticed that the journal writer would let Content data
    go through; Make sure we do not let it do that.

commit 1c9ccb29eea3b974ea0d3613d61a65cd4e83037b
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 9 18:50:18 2020 +0200

    Rename OBJECT_TYPE_KEYS to TEST_OBJECT_DICTS

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/36/ for more details.

This revision is now accepted and ready to land.Apr 10 2020, 4:18 PM

Build is green

Patch application report for D3006 (id=10687)

Could not rebase; Attempt merge onto 9f0cd745d0...

Updating 9f0cd74..63bd07a
Fast-forward
 swh/journal/serializers.py             | 58 ++++++++++++++++++++++++++++++++--
 swh/journal/tests/conftest.py          | 20 ++++++------
 swh/journal/tests/test_kafka_writer.py | 23 +++++++-------
 swh/journal/tests/test_replay.py       | 40 +++++++++++------------
 swh/journal/tests/test_serializers.py  |  9 ++++++
 swh/journal/writer/kafka.py            | 58 ++++++----------------------------
 6 files changed, 114 insertions(+), 94 deletions(-)
Changes applied before test
commit 63bd07af498b4d65d35ea613a1815dc2b51d0a4d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 10 12:45:59 2020 +0200

    Ensure all object keys are passed through unharmed by the journal
    
    The previous implementation of this test only checked keys that were plain
    bytes. It turns out that the "compounded" keys decoding was wrong.
    
    This allows us to simplify the TEST_OBJECT_DICTS structure to only contain a
    list of object dicts.

commit cc5797df4b08c2493155997fcde85e9155e18e31
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Apr 9 17:58:40 2020 +0200

    Move _get_key from the kafka writer to serializers
    
    Use the opportunity to add some trivial smoke tests

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/42/ for more details.