Page MenuHomeSoftware Heritage

idx.storage.JournalWriter: pass value_sanitizer to get_journal_writer.
ClosedPublic

Authored by vlorentz on Nov 27 2020, 1:37 PM.

Details

Summary

It will become mandatory in a future version.

Diff Detail

Repository
rDCIDX Metadata indexer
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D4623 (id=16387)

Could not rebase; Attempt merge onto d87fd53114...

Updating d87fd53..73e181b
Fast-forward
 swh/indexer/storage/model.py            | 15 +++++++++------
 swh/indexer/storage/writer.py           |  4 +++-
 swh/indexer/tests/storage/test_model.py | 26 ++++++++++++++++++++++++++
 3 files changed, 38 insertions(+), 7 deletions(-)
 create mode 100644 swh/indexer/tests/storage/test_model.py
Changes applied before test
commit 73e181b05510ff2325ab4b64a71d64bd7807e899
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Nov 27 13:37:14 2020 +0100

    idx.storage.JournalWriter: pass value_sanitizer to get_journal_writer.
    
    It will become mandatory in a future version.

commit 2cca6cbcf0b4fafaefbb2aa1d90b662a3a8a997d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Nov 27 13:34:05 2020 +0100

    BaseRow.unique_key: Don't crash when indexer_configuration_id is None.
    
    This always happens when writing to Kafka, as the Kafka writer sets it
    to None at the same it it injects the 'tool' data.
    
    This was not caught by tests because they use the in-mem writer;
    which did not call unique_key() at all in swh-journal<=v0.5.1
    (but future versions will).

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/129/ for more details.

This revision is now accepted and ready to land.Nov 27 2020, 1:41 PM
swh/indexer/storage/writer.py
41

That's actually:

**journal_writer, lambda  object_type, dict_value: dict_value

or something.