Page MenuHomeSoftware Heritage

common/utils: Use cache in origin_visit_types function
ClosedPublic

Authored by anlambert on Apr 7 2022, 12:06 PM.

Details

Summary

That function is used to dynamically retrieve the list of origin visit
types (git, svn, npm, ...) currently supported by Software Heritage.

That list of visit types is then used to populate the visit type
dropdown in the search form available in the home and search pages
of the web application.

However getting that visit types list is costly as it involves sending
an elasticsearch query through the use of swh-search.

So instead of sending that costly query each time a search form must
be rendered, prefer to cache the visit types list and update it
every 24 hours.

That change significantly speedup the homepage rendering of the
web application.

Depends on D7526

Diff Detail

Repository
rDWAPPS Web applications
Branch
origin-visit-types-cache
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 28235
Build 44207: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 44206: arc lint + arc unit

Event Timeline

vlorentz accepted this revision.
This revision is now accepted and ready to land.Apr 7 2022, 12:10 PM

Build is green

Patch application report for D7521 (id=27286)

Rebasing onto 17cd03607c...

Current branch diff-target is up to date.
Changes applied before test
commit 5cf94480ff26c91e36176129e868242013e9973f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Apr 7 11:54:51 2022 +0200

    common/utils: Use cache in origin_visit_types function
    
    That function is used to dynamically retrieve the list of origin visit
    types (git, svn, npm, ...) currently supported by Software Heritage.
    
    That list of visit types is then used to populate the visit type
    dropdown in the search form available in the home and search pages
    of the web application.
    
    However getting that visit types list is costly as it involves sending
    an elasticsearch query through the use of swh-search.
    
    So instead of sending that costly query each time a search form must
    be rendered, prefer to cache the visit types list and update it
    every 24 hours.
    
    That change significantly speedup the homepage rendering of the
    web application.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1694/ for more details.

Update: Use django_cache decorator

Build is green

Patch application report for D7521 (id=27305)

Could not rebase; Attempt merge onto 96b27bf43e...

Updating 96b27bf4..bdee9fe4
Fast-forward
 swh/web/browse/snapshot_context.py | 26 +++--------
 swh/web/browse/utils.py            | 32 +++++--------
 swh/web/common/utils.py            | 87 ++++++++++++++++++++++++++--------
 swh/web/misc/coverage.py           | 95 +++++++++++++++++++++-----------------
 swh/web/tests/common/test_utils.py | 76 ++++++++++++++++++++++++++++++
 5 files changed, 215 insertions(+), 101 deletions(-)
Changes applied before test
commit bdee9fe46e31c327a057972e094009ca6c366232
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Apr 7 11:54:51 2022 +0200

    common/utils: Use cache in origin_visit_types function
    
    That function is used to dynamically retrieve the list of origin visit
    types (git, svn, npm, ...) currently supported by Software Heritage.
    
    That list of visit types is then used to populate the visit type
    dropdown in the search form available in the home and search pages
    of the web application.
    
    However getting that visit types list is costly as it involves sending
    an elasticsearch query through the use of swh-search.
    
    So instead of sending that costly query each time a search form must
    be rendered, prefer to cache the visit types list and update it
    every 24 hours.
    
    That change significantly speedup the homepage rendering of the
    web application.

commit b364c93d81a6286c238336aa102b7ba2a55b5d66
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Apr 7 16:59:06 2022 +0200

    common/utils: Add a django_cache decorator and use it in codebase
    
    Add a decorator enabling to put the result of a function call in Django
    cache, subsequent calls will then directly return the cached value.
    
    Use that decorator whenever it is possible in swh-web codebase to simplify
    the handling of cached values and improve code readability.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1697/ for more details.