Page MenuHomeSoftware Heritage

Add a bit of logging in the buffer proxy storage
AcceptedPublic

Authored by douardda on Thu, Apr 29, 3:48 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Summary

Depends in D5646

Diff Detail

Event Timeline

Build is green

Patch application report for D5648 (id=20163)

Could not rebase; Attempt merge onto 92d551a4a5...

Updating 92d551a4..5ab44993
Fast-forward
 swh/storage/buffer.py   | 30 +++++++++++++++++++++++++++---
 swh/storage/filter.py   |  2 --
 swh/storage/validate.py | 34 +++++++++++++++++++++++++++++++++-
 3 files changed, 60 insertions(+), 6 deletions(-)
Changes applied before test
commit 5ab4499301b1ce2e61a52ed55f463a8601c56b9a
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 29 15:45:10 2021 +0200

    Add a bit of logging in the buffer proxy storage

commit c8e5bcdf9b8c5575cd4779e759a69b001cfe1c85
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 29 15:25:08 2021 +0200

    Make buffer and validate proxy storage also handle other object types
    
    but OriginVisit (which requires _add() method normalization).
    
    Also remove unused 'FilteringProxyStorage.object_types' class attribute.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1292/ for more details.

Build is green

Patch application report for D5648 (id=20165)

Could not rebase; Attempt merge onto 92d551a4a5...

Updating 92d551a4..1df03aaf
Fast-forward
 swh/storage/buffer.py   | 45 ++++++++++++++++++++++++++++++++++++++++++---
 swh/storage/filter.py   |  2 --
 swh/storage/validate.py | 34 +++++++++++++++++++++++++++++++++-
 3 files changed, 75 insertions(+), 6 deletions(-)
Changes applied before test
commit 1df03aaf8b3e08cad78e7d0c3dab39b041d2e674
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 29 15:45:10 2021 +0200

    Add a bit of logging in the buffer proxy storage

commit 280386107925c29404c4a8caec48d222ebfb0d01
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 29 15:25:08 2021 +0200

    Make buffer and validate proxy storage also handle other object types
    
    but OriginVisit (which requires _add() method normalization).
    
    Also remove unused 'FilteringProxyStorage.object_types' class attribute.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1294/ for more details.

vlorentz added a subscriber: vlorentz.

What's the motivation?

This revision is now accepted and ready to land.Thu, Apr 29, 4:37 PM

Build is green

Patch application report for D5648 (id=20167)

Could not rebase; Attempt merge onto 92d551a4a5...

Updating 92d551a4..18e6878a
Fast-forward
 swh/storage/buffer.py   | 59 ++++++++++++++++++++++++++++++++++++++++++++++---
 swh/storage/filter.py   |  2 --
 swh/storage/validate.py | 34 +++++++++++++++++++++++++++-
 3 files changed, 89 insertions(+), 6 deletions(-)
Changes applied before test
commit 18e6878ac9a60c649410f086a0c8870d82324bd1
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 29 15:45:10 2021 +0200

    Add a bit of logging in the buffer proxy storage

commit 0161570ec16117df782cded80f6a3704e9aaa39f
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 29 15:25:08 2021 +0200

    Make buffer and validate proxy storage also handle other object types
    
    but OriginVisit (which requires _add() method normalization).
    
    Also remove unused 'FilteringProxyStorage.object_types' class attribute.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1296/ for more details.

What's the motivation?

having a bit of insight on what's going on in a mirroring session, but it ends up that using a buffering proxy for this does not makes much sense, so... i'll probably drop it

it ends up that using a buffering proxy for this does not makes much sense

Er, yeah... It even introduces a bug: unless you explicitly flush before committing to Kafka, some messages won't be replayed if the process crashes. And if you do flush every time, then this proxy is indeed not needed (assuming you use the default journal_client abstraction based on a worker function)