Page MenuHomeSoftware Heritage

Add lru cache to revision parents query function
ClosedPublic

Authored by aeviso on Jan 6 2022, 3:31 PM.

Details

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D6887 (id=24976)

Could not rebase; Attempt merge onto 97816340f6...

Updating 9781634..f8f9c3d
Fast-forward
 swh/provenance/postgresql/archive.py | 94 ++++++++++++++++++++++--------------
 1 file changed, 57 insertions(+), 37 deletions(-)
Changes applied before test
commit f8f9c3dab02fd8982e43a971f3022a0c22fc63c5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jan 6 15:29:53 2022 +0100

    Add lru cache to revision parents query function

commit da626a6edd9ecd5253b606f9f6a3993c713ed411
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jan 6 15:02:56 2022 +0100

    Improve direct sql queries to the archive
    
    Original `directory_ls` query is cleaned up to avoid retrieving
    unnecesary information. Specific query is added for the case where
    no file filtering by size is done.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/559/ for more details.

aeviso requested review of this revision.Jan 6 2022, 3:36 PM
ardumont added a subscriber: ardumont.

lgtm

one suggestion inline.

swh/provenance/postgresql/archive.py
89–90
92

I have no idea what a sensible number would be here ;)

This revision is now accepted and ready to land.Jan 6 2022, 4:02 PM
aeviso added inline comments.
swh/provenance/postgresql/archive.py
92

Me neither, but this one has been working good so far... Maybe there is a way to send stats about hits/misses to StatsD to have better visualization and do some proper experimentation?

Thanks for the review!

aeviso marked an inline comment as done.

rebase

Build is green

Patch application report for D6887 (id=24980)

Rebasing onto da626a6edd...

Current branch diff-target is up to date.
Changes applied before test
commit cc7401096daa57fcb65bb6c3712763faff6485eb
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jan 6 15:29:53 2022 +0100

    Add lru cache to revision parents query function

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/560/ for more details.