Page MenuHomeSoftware Heritage
Differential D6565

Pass the object_type to JournalClient.value_serializer()
ClosedPublic
Actions

Authored by douardda on Oct 27 2021, 4:19 PM.

Details

Reviewers
ardumont
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror)
Commits
rDJNLf92d4acf30e6: Pass the object_type to JournalClient.value_serializer()
Summary

and make this function an (optional) constructor argument.
If not given, stick to kafka_to_value.

This is needed in order to make it possible for the JournalClient to use
a special value_deserializer implementation that needs the object_type,
for example to make the value_deserializer directly instanciate
BaseModel object.

This will be used by an upcoming refactoring of the storage replayer
that will make sure any BaseModel object coming from the journal is valid,
and log invalid kafka objects in case it's not.

Related to T3693.

Depends on D6564.

Diff Detail

Event Timeline

douardda created this revision.Oct 27 2021, 4:19 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 27 2021, 4:19 PM
swh-public-ci added a comment.Oct 27 2021, 4:23 PM

Build is green

Patch application report for D6565 (id=23858)

Could not rebase; Attempt merge onto 95d945e46e...

Updating 95d945e..de483d7
Fast-forward
 docs/journal-clients.rst | 10 +++++++---
 swh/journal/client.py    | 24 +++++++++++++++++-------
 2 files changed, 24 insertions(+), 10 deletions(-)
Changes applied before test
commit de483d7c907e8871ed45859cf09f7ff845d9f247
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 27 14:14:23 2021 +0200

    Pass the object_type to JournalClient.value_serializer()
    
    and make this function an (optional) constructor argument.
    If not given, stick to `kafka_to_value`.
    
    This is needed in order to make it possible for the JournalClient to use
    a special value_deserializer implementation that needs the object_type,
    for example to make the value_deserializer directly instanciate
    BaseModel object.
    
    This will be used by an upcoming refactoring of the storage replayer
    that will make sure any BaseModel object coming from the journal is valid,
    and log invalid kafka objects in case it's not.
    
    Related to T3693.

commit 88054da89653d0e28e6600842eb9287db25b5244
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 27 16:14:17 2021 +0200

    Do call consumer.commit() even if not objects have been received

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/174/ for more details.

Harbormaster completed remote builds in B24741: Diff 23858.Oct 27 2021, 4:23 PM
douardda requested review of this revision.Oct 27 2021, 4:23 PM
ardumont accepted this revision.Oct 27 2021, 5:20 PM
This revision is now accepted and ready to land.Oct 27 2021, 5:20 PM
douardda mentioned this in D6571: Add support for a redis-based reporting for invalid mirrorred objects.Oct 28 2021, 11:24 AM
vlorentz requested changes to this revision.Oct 28 2021, 11:38 AM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/journal/client.py
76–79

You should document the arguments of the function (since its signature is not the same as kafka_to_value)

This revision now requires changes to proceed.Oct 28 2021, 11:38 AM
douardda added inline comments.Oct 28 2021, 11:42 AM
swh/journal/client.py
76–79

Ah yes, I had this in mind then forgot. thx

douardda updated this revision to Diff 23888.Oct 28 2021, 12:11 PM

Document a bit more the value_deserializer and add a test for it

swh-public-ci added a comment.Oct 28 2021, 12:15 PM

Build has FAILED

Patch application report for D6565 (id=23888)

Could not rebase; Attempt merge onto 95d945e46e...

Updating 95d945e..a0c9174
Fast-forward
 docs/journal-clients.rst         | 10 +++++---
 swh/journal/client.py            | 30 ++++++++++++++++++------
 swh/journal/tests/test_client.py | 49 +++++++++++++++++++++++++++++++++++++---
 3 files changed, 76 insertions(+), 13 deletions(-)
Changes applied before test
commit a0c9174d81e3aa3b5c2c1e7446e50446c673e527
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 27 14:14:23 2021 +0200

    Pass the object_type to JournalClient.value_serializer()
    
    and make this function an (optional) constructor argument.
    If not given, stick to `kafka_to_value`.
    
    If the returned value is None, it is ignored (not passed to the
    `worker_fn` function).
    
    This is needed in order to make it possible for the JournalClient to use
    a special value_deserializer implementation that needs the object_type,
    for example to make the value_deserializer directly instanciate
    BaseModel object.
    
    This will be used by an upcoming refactoring of the storage replayer
    that will make sure any BaseModel object coming from the journal is valid,
    and log invalid kafka objects in case it's not.
    
    Related to T3693.

commit 88054da89653d0e28e6600842eb9287db25b5244
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 27 16:14:17 2021 +0200

    Do call consumer.commit() even if not objects have been received

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/175/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/175/console

Harbormaster failed remote builds in B24768: Diff 23888!Oct 28 2021, 12:15 PM
swh-public-ci added a comment.Oct 28 2021, 2:00 PM

Build is green

Patch application report for D6565 (id=23888)

Could not rebase; Attempt merge onto 95d945e46e...

Updating 95d945e..a0c9174
Fast-forward
 docs/journal-clients.rst         | 10 +++++---
 swh/journal/client.py            | 30 ++++++++++++++++++------
 swh/journal/tests/test_client.py | 49 +++++++++++++++++++++++++++++++++++++---
 3 files changed, 76 insertions(+), 13 deletions(-)
Changes applied before test
commit a0c9174d81e3aa3b5c2c1e7446e50446c673e527
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 27 14:14:23 2021 +0200

    Pass the object_type to JournalClient.value_serializer()
    
    and make this function an (optional) constructor argument.
    If not given, stick to `kafka_to_value`.
    
    If the returned value is None, it is ignored (not passed to the
    `worker_fn` function).
    
    This is needed in order to make it possible for the JournalClient to use
    a special value_deserializer implementation that needs the object_type,
    for example to make the value_deserializer directly instanciate
    BaseModel object.
    
    This will be used by an upcoming refactoring of the storage replayer
    that will make sure any BaseModel object coming from the journal is valid,
    and log invalid kafka objects in case it's not.
    
    Related to T3693.

commit 88054da89653d0e28e6600842eb9287db25b5244
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 27 16:14:17 2021 +0200

    Do call consumer.commit() even if not objects have been received

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/176/ for more details.

Harbormaster completed remote builds in B24768: Diff 23888.Oct 28 2021, 2:00 PM
douardda updated this revision to Diff 23899.Oct 28 2021, 2:03 PM

typo

swh-public-ci added a comment.Oct 28 2021, 2:07 PM

Build is green

Patch application report for D6565 (id=23899)

Could not rebase; Attempt merge onto 95d945e46e...

Updating 95d945e..e267ee1
Fast-forward
 docs/journal-clients.rst         | 10 +++++---
 swh/journal/client.py            | 30 ++++++++++++++++++------
 swh/journal/tests/test_client.py | 49 +++++++++++++++++++++++++++++++++++++---
 3 files changed, 76 insertions(+), 13 deletions(-)
Changes applied before test
commit e267ee178ccc51a180d75eba2e319d23d1a9d6bb
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 27 14:14:23 2021 +0200

    Pass the object_type to JournalClient.value_serializer()
    
    and make this function an (optional) constructor argument.
    If not given, stick to `kafka_to_value`.
    
    If the returned value is None, it is ignored (not passed to the
    `worker_fn` function).
    
    This is needed in order to make it possible for the JournalClient to use
    a special value_deserializer implementation that needs the object_type,
    for example to make the value_deserializer directly instanciate
    BaseModel object.
    
    This will be used by an upcoming refactoring of the storage replayer
    that will make sure any BaseModel object coming from the journal is valid,
    and log invalid kafka objects in case it's not.
    
    Related to T3693.

commit 88054da89653d0e28e6600842eb9287db25b5244
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 27 16:14:17 2021 +0200

    Do call consumer.commit() even if not objects have been received

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/177/ for more details.

Harbormaster completed remote builds in B24779: Diff 23899.Oct 28 2021, 2:07 PM
douardda updated this revision to Diff 23900.Oct 28 2021, 2:08 PM

^x^s...

swh-public-ci added a comment.Oct 28 2021, 2:12 PM

Build is green

Patch application report for D6565 (id=23900)

Could not rebase; Attempt merge onto 95d945e46e...

Updating 95d945e..f92d4ac
Fast-forward
 docs/journal-clients.rst         | 10 +++++---
 swh/journal/client.py            | 31 +++++++++++++++++++------
 swh/journal/tests/test_client.py | 49 +++++++++++++++++++++++++++++++++++++---
 3 files changed, 77 insertions(+), 13 deletions(-)
Changes applied before test
commit f92d4acf30e64bd96e4fd536605ddb943bdbd369
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 27 14:14:23 2021 +0200

    Pass the object_type to JournalClient.value_serializer()
    
    and make this function an (optional) constructor argument.
    If not given, stick to `kafka_to_value`.
    
    If the returned value is None, it is ignored (not passed to the
    `worker_fn` function).
    
    This is needed in order to make it possible for the JournalClient to use
    a special value_deserializer implementation that needs the object_type,
    for example to make the value_deserializer directly instanciate
    BaseModel object.
    
    This will be used by an upcoming refactoring of the storage replayer
    that will make sure any BaseModel object coming from the journal is valid,
    and log invalid kafka objects in case it's not.
    
    Related to T3693.

commit 88054da89653d0e28e6600842eb9287db25b5244
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 27 16:14:17 2021 +0200

    Do call consumer.commit() even if not objects have been received

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/178/ for more details.

Harbormaster completed remote builds in B24780: Diff 23900.Oct 28 2021, 2:12 PM
vlorentz accepted this revision.Oct 28 2021, 2:12 PM
This revision is now accepted and ready to land.Oct 28 2021, 2:12 PM
Closed by commit rDJNLf92d4acf30e6: Pass the object_type to JournalClient.value_serializer() (authored by douardda). · Explain WhyOct 28 2021, 2:34 PM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDJNLf92d4acf30e6: Pass the object_type to JournalClient.value_serializer().

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
docs/
10 lines--
swh/
journal/
22 lines80%
Loading...

Diff 23858

View Options

docs/journal-clients.rst

Loading...
View Options

swh/journal/client.py

Loading...
About · Developer information · Legal