Page MenuHomeSoftware Heritage
Differential D7224

misc/coverage: Use django cache instead of LRU ones
ClosedPublic
Actions

Authored by anlambert on Feb 22 2022, 5:27 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDWAPPS7b917f18fd15: misc/coverage: Use django cache instead of LRU ones
Summary

It enables to update origin counters more frequently in the coverage
widget of the production webapp.

Also disable cache use when running swh-web in development or staging
environment to get accurate counters on each homepage refresh.

Depends on D7223

Diff Detail

Repository
rDWAPPS Web applications
Branch
coverage-replace-lru-cache-by-django-cache
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 27072
Build 42337: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 42336: arc lint + arc unit

Event Timeline

anlambert created this revision.Feb 22 2022, 5:27 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 22 2022, 5:27 PM
swh-public-ci added a comment.Feb 22 2022, 5:44 PM

Build is green

Patch application report for D7224 (id=26182)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..cb5f4ca9
Fast-forward
 swh/web/common/utils.py             | 40 +++++++++++++++++++++++++++----------
 swh/web/config.py                   |  5 +++--
 swh/web/misc/coverage.py            | 21 +++++++++++++++----
 swh/web/tests/common/test_utils.py  | 19 +++++++++++++++++-
 swh/web/tests/misc/test_coverage.py | 17 ++++++++++++----
 swh/web/tests/test_templates.py     | 11 +++++++---
 6 files changed, 89 insertions(+), 24 deletions(-)
Changes applied before test
commit cb5f4ca9386386b70c529048a739b1f4e7a67962
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Bypass LRU caches when not in production
    
    It enables to get accurate origin counters when executing swh-web
    in a development environment or in staging.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1399/ for more details.

Harbormaster completed remote builds in B27054: Diff 26182.Feb 22 2022, 5:44 PM
anlambert requested review of this revision.Feb 22 2022, 5:44 PM
anlambert updated this revision to Diff 26184.Feb 22 2022, 5:50 PM

Fix typo in comment

vlorentz added a subscriber: vlorentz.Feb 22 2022, 5:58 PM

This code feels rather weird. I think you should get rid of the lru_cache decorator on _get_listers_metrics, and implement caching directly in _get_listers_metrics, with a boolean argument to toggle it.

vlorentz requested changes to this revision.Feb 22 2022, 5:58 PM
This revision now requires changes to proceed.Feb 22 2022, 5:58 PM
swh-public-ci added a comment.Feb 22 2022, 6:07 PM

Build is green

Patch application report for D7224 (id=26184)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..880e4e1f
Fast-forward
 swh/web/common/utils.py             | 40 +++++++++++++++++++++++++++----------
 swh/web/config.py                   |  5 +++--
 swh/web/misc/coverage.py            | 21 +++++++++++++++----
 swh/web/tests/common/test_utils.py  | 19 +++++++++++++++++-
 swh/web/tests/misc/test_coverage.py | 17 ++++++++++++----
 swh/web/tests/test_templates.py     | 11 +++++++---
 6 files changed, 89 insertions(+), 24 deletions(-)
Changes applied before test
commit 880e4e1f60fd269b4f443859ebb2d6f366609212
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Bypass LRU caches when not in production
    
    It enables to get accurate origin counters when executing swh-web
    in a development environment or in staging.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1400/ for more details.

Harbormaster completed remote builds in B27056: Diff 26184.Feb 22 2022, 6:07 PM
anlambert added a comment.Feb 22 2022, 7:14 PM
In D7224#188075, @vlorentz wrote:

This code feels rather weird. I think you should get rid of the lru_cache decorator on _get_listers_metrics, and implement caching directly in _get_listers_metrics, with a boolean argument to toggle it.

I could use django cache instead, this will also have the advantage of resetting the cache on a regular basis and update counters more frequently.

anlambert updated this revision to Diff 26187.Feb 22 2022, 7:49 PM

Use django cache instead of LRU caches.

anlambert retitled this revision from misc/coverage: Bypass LRU caches when not in production to misc/coverage: misc/coverage: Use django cache instead of LRU ones.Feb 22 2022, 7:50 PM
anlambert edited the summary of this revision. (Show Details)
anlambert retitled this revision from misc/coverage: misc/coverage: Use django cache instead of LRU ones to misc/coverage: Use django cache instead of LRU ones.
anlambert updated this revision to Diff 26188.Feb 22 2022, 7:52 PM

Update comment in test

swh-public-ci added a comment.Feb 22 2022, 8:08 PM

Build has FAILED

Patch application report for D7224 (id=26187)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..77856583
Fast-forward
 swh/web/common/utils.py             |  40 +++++++++---
 swh/web/config.py                   |   5 +-
 swh/web/misc/coverage.py            | 117 ++++++++++++++++++++++--------------
 swh/web/tests/common/test_utils.py  |  19 +++++-
 swh/web/tests/misc/test_coverage.py |  33 +++++-----
 swh/web/tests/test_templates.py     |  11 +++-
 6 files changed, 144 insertions(+), 81 deletions(-)
Changes applied before test
commit 77856583981ce1080c19bd11c28d6bdce8c6ce86
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Use django cache instead of LRU ones
    
    It enables to update origin counters more frequently in the coverage
    widget of the production webapp.
    
    Also disable cache use when running swh-web in development or staging
    environment to get accurate counters on each homepage refresh.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1401/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1401/console

Harbormaster failed remote builds in B27059: Diff 26187!Feb 22 2022, 8:08 PM
swh-public-ci added a comment.Feb 22 2022, 8:09 PM

Build has FAILED

Patch application report for D7224 (id=26188)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..4d14f4b3
Fast-forward
 swh/web/common/utils.py             |  40 +++++++++---
 swh/web/config.py                   |   5 +-
 swh/web/misc/coverage.py            | 117 ++++++++++++++++++++++--------------
 swh/web/tests/common/test_utils.py  |  19 +++++-
 swh/web/tests/misc/test_coverage.py |  33 +++++-----
 swh/web/tests/test_templates.py     |  11 +++-
 6 files changed, 144 insertions(+), 81 deletions(-)
Changes applied before test
commit 4d14f4b335948954bce0e8054bb6c6d35e48bda4
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Use django cache instead of LRU ones
    
    It enables to update origin counters more frequently in the coverage
    widget of the production webapp.
    
    Also disable cache use when running swh-web in development or staging
    environment to get accurate counters on each homepage refresh.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1402/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1402/console

Harbormaster failed remote builds in B27060: Diff 26188!Feb 22 2022, 8:10 PM
vlorentz accepted this revision.Feb 23 2022, 10:52 AM

great, thanks!

This revision is now accepted and ready to land.Feb 23 2022, 10:52 AM
anlambert edited the summary of this revision. (Show Details)Feb 23 2022, 11:12 AM
anlambert updated this revision to Diff 26199.Feb 23 2022, 11:25 AM

Return adequate default value when value is not in cache.

swh-public-ci added a comment.Feb 23 2022, 11:27 AM

Build has FAILED

Patch application report for D7224 (id=26199)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..7b917f18
Fast-forward
 swh/web/common/utils.py             |  40 +++++++++---
 swh/web/config.py                   |   5 +-
 swh/web/misc/coverage.py            | 117 ++++++++++++++++++++++--------------
 swh/web/tests/common/test_utils.py  |  19 +++++-
 swh/web/tests/misc/test_coverage.py |  33 +++++-----
 swh/web/tests/test_templates.py     |  11 +++-
 6 files changed, 144 insertions(+), 81 deletions(-)
Changes applied before test
commit 7b917f18fd155e8d76f2df6edb0c5b72aabe8530
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Use django cache instead of LRU ones
    
    It enables to update origin counters more frequently in the coverage
    widget of the production webapp.
    
    Also disable cache use when running swh-web in development or staging
    environment to get accurate counters on each homepage refresh.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1403/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1403/console

Harbormaster failed remote builds in B27072: Diff 26199!Feb 23 2022, 11:27 AM
swh-public-ci added a comment.Feb 23 2022, 11:30 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1404/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1404/console

swh-public-ci added a comment.Feb 23 2022, 11:45 AM

Build is green

Patch application report for D7224 (id=26199)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..7b917f18
Fast-forward
 swh/web/common/utils.py             |  40 +++++++++---
 swh/web/config.py                   |   5 +-
 swh/web/misc/coverage.py            | 117 ++++++++++++++++++++++--------------
 swh/web/tests/common/test_utils.py  |  19 +++++-
 swh/web/tests/misc/test_coverage.py |  33 +++++-----
 swh/web/tests/test_templates.py     |  11 +++-
 6 files changed, 144 insertions(+), 81 deletions(-)
Changes applied before test
commit 7b917f18fd155e8d76f2df6edb0c5b72aabe8530
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Use django cache instead of LRU ones
    
    It enables to update origin counters more frequently in the coverage
    widget of the production webapp.
    
    Also disable cache use when running swh-web in development or staging
    environment to get accurate counters on each homepage refresh.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1405/ for more details.

Harbormaster completed remote builds in B27072: Diff 26199.Feb 23 2022, 11:45 AM
Closed by commit rDWAPPS7b917f18fd15: misc/coverage: Use django cache instead of LRU ones (authored by anlambert). · Explain WhyFeb 23 2022, 1:07 PM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDWAPPS7b917f18fd15: misc/coverage: Use django cache instead of LRU ones.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
web/
misc/
117 lines97%
Loading...
tests/
misc/
33 lines100%
Loading...

Diff 26199

View Options

swh/web/misc/coverage.py

Loading...
View Options

swh/web/tests/misc/test_coverage.py

Loading...
About · Developer information · Legal