Page MenuHomeSoftware Heritage

Avoid flushing empty buffers in the `Provenance` backend
ClosedPublic

Authored by aeviso on Sep 7 2021, 4:52 PM.

Details

Summary

Also, remove the use of f-strings when logging to avoid unnecessary
parameter resolution.

Remove old debug logging and improve other's messages

Depends on D6205

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D6208 (id=22471)

Could not rebase; Attempt merge onto 759d39a44c...

Updating 759d39a..26061a0
Fast-forward
 swh/provenance/graph.py                 |  15 ----
 swh/provenance/mongo/backend.py         |  12 +--
 swh/provenance/origin.py                |  10 ---
 swh/provenance/postgresql/provenance.py |  10 ++-
 swh/provenance/provenance.py            | 147 +++++++++++++++++---------------
 swh/provenance/revision.py              |  13 ---
 6 files changed, 89 insertions(+), 118 deletions(-)
Changes applied before test
commit 26061a0a15a0dd1b4694b27750a84e71b79feea9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:32:55 2021 +0200

    Remove old debug logging and improve other's messages

commit 00f6d9acfb2ef5ce0527ad2bf625457063f808ba
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:10:28 2021 +0200

    Avoid flushing empty buffers in the `Provenance` backend
    
    Also, remove the use of f-strings when logging to avoid unnecessary
    parameter resolution.

commit bd6de9eaba198199536df2677bee0332af02f8b3
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 3 16:21:58 2021 +0200

    Fix two minor bugs on the MongoDB backend

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/322/ for more details.

aeviso requested review of this revision.Sep 7 2021, 4:55 PM

I'm not too sure about the flat-out removal of a lot of these logging statements (rather than just turning them into logging statements with deferred formatting), but the changes to the flush() logic at least look sound.

In D6208#160587, @olasd wrote:

I'm not too sure about the flat-out removal of a lot of these logging statements (rather than just turning them into logging statements with deferred formatting), but the changes to the flush() logic at least look sound.

Which ones do you believe are worth keeping? The ones being removed are just debug messages I used for some specific tested in the past and are not really needed anymore. Most of them are quite silly and provide very few (or almost no) information of what's going on. The only ones that might be worth keeping are those logging execution time for the two main methods in the module (revision_add and origin_add) but we plan to add some statistics using statsd to have better visualization of what's going on inside the module.

ACK. These were the ones I was indeed thinking of. All good then :-)

This revision is now accepted and ready to land.Sep 7 2021, 5:42 PM