Page MenuHomeSoftware Heritage

misc/coverage: Use django cache instead of LRU ones
ClosedPublic

Authored by anlambert on Feb 22 2022, 5:27 PM.

Details

Summary

It enables to update origin counters more frequently in the coverage
widget of the production webapp.

Also disable cache use when running swh-web in development or staging
environment to get accurate counters on each homepage refresh.

Depends on D7223

Diff Detail

Repository
rDWAPPS Web applications
Branch
coverage-replace-lru-cache-by-django-cache
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 27072
Build 42337: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 42336: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D7224 (id=26182)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..cb5f4ca9
Fast-forward
 swh/web/common/utils.py             | 40 +++++++++++++++++++++++++++----------
 swh/web/config.py                   |  5 +++--
 swh/web/misc/coverage.py            | 21 +++++++++++++++----
 swh/web/tests/common/test_utils.py  | 19 +++++++++++++++++-
 swh/web/tests/misc/test_coverage.py | 17 ++++++++++++----
 swh/web/tests/test_templates.py     | 11 +++++++---
 6 files changed, 89 insertions(+), 24 deletions(-)
Changes applied before test
commit cb5f4ca9386386b70c529048a739b1f4e7a67962
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Bypass LRU caches when not in production
    
    It enables to get accurate origin counters when executing swh-web
    in a development environment or in staging.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1399/ for more details.

This code feels rather weird. I think you should get rid of the lru_cache decorator on _get_listers_metrics, and implement caching directly in _get_listers_metrics, with a boolean argument to toggle it.

This revision now requires changes to proceed.Feb 22 2022, 5:58 PM

Build is green

Patch application report for D7224 (id=26184)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..880e4e1f
Fast-forward
 swh/web/common/utils.py             | 40 +++++++++++++++++++++++++++----------
 swh/web/config.py                   |  5 +++--
 swh/web/misc/coverage.py            | 21 +++++++++++++++----
 swh/web/tests/common/test_utils.py  | 19 +++++++++++++++++-
 swh/web/tests/misc/test_coverage.py | 17 ++++++++++++----
 swh/web/tests/test_templates.py     | 11 +++++++---
 6 files changed, 89 insertions(+), 24 deletions(-)
Changes applied before test
commit 880e4e1f60fd269b4f443859ebb2d6f366609212
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Bypass LRU caches when not in production
    
    It enables to get accurate origin counters when executing swh-web
    in a development environment or in staging.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1400/ for more details.

This code feels rather weird. I think you should get rid of the lru_cache decorator on _get_listers_metrics, and implement caching directly in _get_listers_metrics, with a boolean argument to toggle it.

I could use django cache instead, this will also have the advantage of resetting the cache on a regular basis and update counters more frequently.

Use django cache instead of LRU caches.

anlambert retitled this revision from misc/coverage: Bypass LRU caches when not in production to misc/coverage: misc/coverage: Use django cache instead of LRU ones.Feb 22 2022, 7:50 PM
anlambert edited the summary of this revision. (Show Details)
anlambert retitled this revision from misc/coverage: misc/coverage: Use django cache instead of LRU ones to misc/coverage: Use django cache instead of LRU ones.

Build has FAILED

Patch application report for D7224 (id=26187)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..77856583
Fast-forward
 swh/web/common/utils.py             |  40 +++++++++---
 swh/web/config.py                   |   5 +-
 swh/web/misc/coverage.py            | 117 ++++++++++++++++++++++--------------
 swh/web/tests/common/test_utils.py  |  19 +++++-
 swh/web/tests/misc/test_coverage.py |  33 +++++-----
 swh/web/tests/test_templates.py     |  11 +++-
 6 files changed, 144 insertions(+), 81 deletions(-)
Changes applied before test
commit 77856583981ce1080c19bd11c28d6bdce8c6ce86
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Use django cache instead of LRU ones
    
    It enables to update origin counters more frequently in the coverage
    widget of the production webapp.
    
    Also disable cache use when running swh-web in development or staging
    environment to get accurate counters on each homepage refresh.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1401/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1401/console

Build has FAILED

Patch application report for D7224 (id=26188)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..4d14f4b3
Fast-forward
 swh/web/common/utils.py             |  40 +++++++++---
 swh/web/config.py                   |   5 +-
 swh/web/misc/coverage.py            | 117 ++++++++++++++++++++++--------------
 swh/web/tests/common/test_utils.py  |  19 +++++-
 swh/web/tests/misc/test_coverage.py |  33 +++++-----
 swh/web/tests/test_templates.py     |  11 +++-
 6 files changed, 144 insertions(+), 81 deletions(-)
Changes applied before test
commit 4d14f4b335948954bce0e8054bb6c6d35e48bda4
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Use django cache instead of LRU ones
    
    It enables to update origin counters more frequently in the coverage
    widget of the production webapp.
    
    Also disable cache use when running swh-web in development or staging
    environment to get accurate counters on each homepage refresh.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1402/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1402/console

This revision is now accepted and ready to land.Feb 23 2022, 10:52 AM

Return adequate default value when value is not in cache.

Build has FAILED

Patch application report for D7224 (id=26199)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..7b917f18
Fast-forward
 swh/web/common/utils.py             |  40 +++++++++---
 swh/web/config.py                   |   5 +-
 swh/web/misc/coverage.py            | 117 ++++++++++++++++++++++--------------
 swh/web/tests/common/test_utils.py  |  19 +++++-
 swh/web/tests/misc/test_coverage.py |  33 +++++-----
 swh/web/tests/test_templates.py     |  11 +++-
 6 files changed, 144 insertions(+), 81 deletions(-)
Changes applied before test
commit 7b917f18fd155e8d76f2df6edb0c5b72aabe8530
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Use django cache instead of LRU ones
    
    It enables to update origin counters more frequently in the coverage
    widget of the production webapp.
    
    Also disable cache use when running swh-web in development or staging
    environment to get accurate counters on each homepage refresh.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1403/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1403/console

Build is green

Patch application report for D7224 (id=26199)

Could not rebase; Attempt merge onto c14cfd1b28...

Updating c14cfd1b..7b917f18
Fast-forward
 swh/web/common/utils.py             |  40 +++++++++---
 swh/web/config.py                   |   5 +-
 swh/web/misc/coverage.py            | 117 ++++++++++++++++++++++--------------
 swh/web/tests/common/test_utils.py  |  19 +++++-
 swh/web/tests/misc/test_coverage.py |  33 +++++-----
 swh/web/tests/test_templates.py     |  11 +++-
 6 files changed, 144 insertions(+), 81 deletions(-)
Changes applied before test
commit 7b917f18fd155e8d76f2df6edb0c5b72aabe8530
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 17:10:51 2022 +0100

    misc/coverage: Use django cache instead of LRU ones
    
    It enables to update origin counters more frequently in the coverage
    widget of the production webapp.
    
    Also disable cache use when running swh-web in development or staging
    environment to get accurate counters on each homepage refresh.

commit cc5736263984312914af673831932af01f6779eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Feb 22 16:26:14 2022 +0100

    common/utils: Add utility functions to check swh-web run environment
    
    Add functions to check if we are running a development, staging or
    production version of swh-web.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1405/ for more details.