Instead of being a method of KafkaJournalWriter.
This will allow using JournalWriters for values that aren't model objects.
Depends on D4079
Details
Details
- Reviewers
olasd - Group Reviewers
Reviewers - Maniphest Tasks
- T2651: Make the indexer-storage publish its rows to Kafka
- Commits
- rDJNL3ae9a5bab459: Make value_sanitizer an argument of JournalWriter.
Diff Detail
Diff Detail
- Repository
- rDJNL Journal infrastructure
- Branch
- master
- Lint
No Linters Available - Unit
No Unit Test Coverage - Build Status
Buildable 15692 Build 24163: Phabricator diff pipeline on jenkins Jenkins console · Jenkins Build 24162: arc lint + arc unit
Event Timeline
Comment Actions
Build has FAILED
Patch application report for D4080 (id=14398)
Could not rebase; Attempt merge onto 7815cad28a...
Updating 7815cad..dd04112 Fast-forward swh/journal/pytest_plugin.py | 4 +- swh/journal/serializers.py | 80 +--------------------------------- swh/journal/tests/test_kafka_writer.py | 9 +++- swh/journal/tests/test_serializers.py | 11 +---- swh/journal/writer/__init__.py | 12 +++++ swh/journal/writer/inmemory.py | 7 ++- swh/journal/writer/kafka.py | 20 +++------ 7 files changed, 38 insertions(+), 105 deletions(-)
Changes applied before test
commit dd04112fe6c6149d1e3e0a4ff1764c0ae7c3902f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Tue Sep 29 15:06:42 2020 +0200
Make object_sanitizer an argument of JournalWriter.
Instead of being a method of KafkaJournalWriter.
This will allow using JournalWriters for values that aren't model objects.
commit 1a64c96f97d8b6a56a873a269af88859083359ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Tue Sep 29 14:15:35 2020 +0200
Remove swh.journal.serializers.object_key, use BaseModel.unique_key instead.Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/112/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/112/console
Comment Actions
Build has FAILED
Patch application report for D4080 (id=14399)
Could not rebase; Attempt merge onto 7815cad28a...
Updating 7815cad..549624e Fast-forward swh/journal/pytest_plugin.py | 4 +- swh/journal/serializers.py | 80 +--------------------------------- swh/journal/tests/test_kafka_writer.py | 9 +++- swh/journal/tests/test_serializers.py | 11 +---- swh/journal/writer/__init__.py | 12 +++++ swh/journal/writer/inmemory.py | 7 ++- swh/journal/writer/kafka.py | 20 +++------ 7 files changed, 38 insertions(+), 105 deletions(-)
Changes applied before test
commit 549624eff1ee246f0f8552491f730123520376e8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Tue Sep 29 15:06:42 2020 +0200
Make value_sanitizer an argument of JournalWriter.
Instead of being a method of KafkaJournalWriter.
This will allow using JournalWriters for values that aren't model objects.
commit 1a64c96f97d8b6a56a873a269af88859083359ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Tue Sep 29 14:15:35 2020 +0200
Remove swh.journal.serializers.object_key, use BaseModel.unique_key instead.Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/113/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/113/console
Comment Actions
Build is green
Patch application report for D4080 (id=14927)
Could not rebase; Attempt merge onto d3c09acb40...
Merge made by the 'recursive' strategy. requirements-swh.txt | 2 +- swh/journal/pytest_plugin.py | 4 +- swh/journal/serializers.py | 80 +--------------------------------- swh/journal/tests/test_kafka_writer.py | 9 +++- swh/journal/tests/test_serializers.py | 11 +---- swh/journal/writer/__init__.py | 12 +++++ swh/journal/writer/inmemory.py | 7 ++- swh/journal/writer/kafka.py | 20 +++------ 8 files changed, 39 insertions(+), 106 deletions(-)
Changes applied before test
commit cfbd0d8c7aab38a29bc5d74c315da451b56291b2
Merge: d3c09ac 0807bda
Author: Jenkins user <jenkins@localhost>
Date: Mon Oct 12 10:21:04 2020 +0000
Merge branch 'diff-target' into HEAD
commit 0807bda42b4adc4c4f9ba33a417da034bb6142e2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Tue Sep 29 15:06:42 2020 +0200
Make value_sanitizer an argument of JournalWriter.
Instead of being a method of KafkaJournalWriter.
This will allow using JournalWriters for values that aren't model objects.
commit 1f6621ea669f088a1ed0bb5400158038b86d50c5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Tue Sep 29 14:15:35 2020 +0200
Remove swh.journal.serializers.object_key, use BaseModel.unique_key instead.See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/116/ for more details.
Comment Actions
Build is green
Patch application report for D4080 (id=15205)
Rebasing onto abc42e2bf7...
Current branch diff-target is up to date.
Changes applied before test
commit efd94b05bebbfc082227719ac94b2223723f95ed
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Tue Sep 29 15:06:42 2020 +0200
Make value_sanitizer an argument of JournalWriter.
Instead of being a method of KafkaJournalWriter.
This will allow using JournalWriters for values that aren't model objects.See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/119/ for more details.
Comment Actions
I'm not a fan of the sanitizer name (I know I probably introduced it :P). Something along the lines of object_filter may be more appropriate?
If you want to make this completely generic, the calls to to_dict should probably not be inlined to the _write_addition function.
This also feels like, if this becomes configurable, the default value should probably be an identity function rather than a function specific to swh.model.model objects. Of course the default should change once swh.storage explicitly calls the journal_writer initialization pointing to the new function.
Comment Actions
object_filter sounds like it would remove some objects (like Python's filter() function)
If you want to make this completely generic, the calls to to_dict should probably not be inlined to the _write_addition function.
See D4082, which introduces a ValueProtocol that includes to_dict and others.
This also feels like, if this becomes configurable, the default value should probably be an identity function rather than a function specific to swh.model.model objects.
Of course the default should change once swh.storage explicitly calls the journal_writer initialization pointing to the new function.
So after this diff is landed and released, yes.
Comment Actions
Build is green
Patch application report for D4080 (id=15549)
Rebasing onto e998407120...
Current branch diff-target is up to date.
Changes applied before test
commit 3ae9a5bab45970217d4cd93785c47ffcd936ed57
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Tue Sep 29 15:06:42 2020 +0200
Make value_sanitizer an argument of JournalWriter.
Instead of being a method of KafkaJournalWriter.
This will allow using JournalWriters for values that aren't model objects.See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/125/ for more details.