Page MenuHomeSoftware Heritage

prevent high memory usage
ClosedPublic

Authored by kalpitk on Mar 26 2019, 7:08 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Summary

Heap permutation for more than 8 strings takes high amount of memory

Diff Detail

Repository
rDWAPPS Web applications
Branch
search-bug
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 4842
Build 6453: tox-on-jenkinsJenkins
Build 6452: arc lint + arc unit

Event Timeline

My laptop crashed when I entered a big query on /browse/search/ , because it filled my whole RAM.
Array size of 10 filled my 4 GBs of RAM when heapsPermute was called

anlambert requested changes to this revision.EditedMar 26 2019, 7:49 PM
anlambert added a subscriber: anlambert.

Indeed, permutation algorithm is not of optimal complexity so we should avoid its use when the number of input keywords is high.

I would rather restrict the maximum number of keywords to 6 as search queries can also end up with errors
easily starting 7 keywords, see below (used keywords: python django template hello world foo test):

"GET /browse/origin/search/python.*django.*template.*hello.*world.*foo|django.*python.*template.*hello.*world.*foo|template.*python.*django.*hello.*world.*foo|python.*template.*django.*hello.*world.*foo|django.*template.*python.*hello.*world.*foo|template.*django.*python.*hello.*world.*foo|hello.*django.*template.*python.*world.*foo|django.*hello.*template.*python.*world.*foo|template.*hello.*django.*python.*world.*foo|hello.*template.*django.*python.*world.*foo|django.*template.*hello.*python.*world.*foo|template.*django.*hello.*python.*world.*foo|hello.*python.*template.*django.*world.*foo|python.*hello.*template.*django.*world.*foo|template.*hello.*python.*django.*world.*foo|hello.*template.*python.*django.*world.*foo|python.*template.*hello.*django.*world.*foo|template.*python.*hello.*django.*world.*foo|hello.*python.*django.*template.*world.*foo|python.*hello.*django.*template.*world.*foo|django.*hello.*python.*template.*world.*foo|hello.*django.*python.*template.*world.*foo|python.*django.*hello.*template.*world.*foo|django.*python.*hello.*template.*world.*foo|world.*python.*django.*template.*hello.*foo|python.*world.*django.*template.*hello.*foo|django.*world.*python.*template.*hello.*foo|world.*django.*python.*template.*hello.*foo|python.*django.*world.*template.*hello.*foo|django.*python.*world.*template.*hello.*foo|template.*python.*django.*world.*hello.*foo|python.*template.*django.*world.*hello.*foo|django.*template.*python.*world.*hello.*foo|template.*django.*python.*world.*hello.*foo|python.*django.*template.*world.*hello.*foo|django.*python.*template.*world.*hello.*foo|template.*world.*django.*python.*hello.*foo|world.*template.*django.*python.*hello.*foo|django.*template.*world.*python.*hello.*foo|template.*django.*world.*python.*hello.*foo|world.*django.*template.*python.*hello.*foo|django.*world.*template.*python.*hello.*foo|template.*world.*python.*django.*hello.*foo|world.*template.*python.*django.*hello.*foo|python.*template.*world.*django.*hello.*foo|template.*python.*world.*django.*hello.*foo|world.*python.*template.*django.*hello.*foo|python.*world.*template.*django.*hello.*foo|hello.*world.*python.*django.*template.*foo|world.*hello.*python.*django.*template.*foo|python.*hello.*world.*django.*template.*foo|hello.*python.*world.*django.*template.*foo|world.*python.*hello.*django.*template.*foo|python.*world.*hello.*django.*template.*foo|django.*world.*python.*hello.*template.*foo|world.*django.*python.*hello.*template.*foo|python.*django.*world.*hello.*template.*foo|django.*python.*world.*hello.*template.*foo|world.*python.*django.*hello.*template.*foo|python.*world.*django.*hello.*template.*foo|django.*hello.*python.*world.*template.*foo|hello.*django.*python.*world.*template.*foo|python.*django.*hello.*world.*template.*foo|django.*python.*hello.*world.*template.*foo|hello.*python.*django.*world.*template.*foo|python.*hello.*django.*world.*template.*foo|django.*hello.*world.*python.*template.*foo|hello.*django.*world.*python.*template.*foo|world.*django.*hello.*python.*template.*foo|django.*world.*hello.*python.*template.*foo|hello.*world.*django.*python.*template.*foo|world.*hello.*django.*python.*template.*foo|template.*hello.*world.*python.*django.*foo|hello.*template.*world.*python.*django.*foo|world.*template.*hello.*python.*django.*foo|template.*world.*hello.*python.*django.*foo|hello.*world.*template.*python.*django.*foo|world.*hello.*template.*python.*django.*foo|python.*hello.*world.*template.*django.*foo|hello.*python.*world.*template.*django.*foo|world.*python.*hello.*template.*django.*foo|python.*world.*hello.*template.*django.*foo|hello.*world.*python.*template.*django.*foo|world.*hello.*python.*template.*django.*foo|python.*template.*world.*hello.*django.*foo|template.*python.*world.*hello.*django.*foo|world.*python.*template.*hello.*django.*foo|python.*world.*template.*hello.*django.*foo|template.*world.*python.*hello.*django.*foo|world.*template.*python.*hello.*django.*foo|python.*template.*hello.*world.*django.*foo|template.*python.*hello.*world.*django.*foo|hello.*python.*template.*world.*django.*foo|python.*hello.*template.*world.*django.*foo|template.*hello.*python.*world.*django.*foo|hello.*template.*python.*world.*django.*foo|django.*template.*hello.*world.*python.*foo|template.*django.*hello.*world.*python.*foo|hello.*django.*template.*world.*python.*foo|django.*hello.*template.*world.*python.*foo|template.*hello.*django.*world.*python.*foo|hello.*template.*django.*world.*python.*foo|world.*template.*hello.*django.*python.*foo|template.*world.*hello.*django.*python.*foo|hello.*world.*template.*django.*python.*foo|world.*hello.*template.*django.*python.*foo|template.*hello.*world.*django.*python.*foo|hello.*template.*world.*django.*python.*foo|world.*django.*hello.*template.*python.*foo|django.*world.*hello.*template.*python.*foo|hello.*world.*django.*template.*python.*foo|world.*hello.*django.*template.*python.*foo|django.*hello.*world.*template.*python.*foo|hello.*django.*world.*template.*python.*foo|world.*django.*template.*hello.*python.*foo|django.*world.*template.*hello.*python.*foo|template.*world.*django.*hello.*python.*foo|world.*template.*django.*hello.*python.*foo|django.*template.*world.*hello.*python.*foo|template.*django.*world.*hello.*python.*foo|foo.*django.*template.*hello.*world.*python|django.*foo.*template.*hello.*world.*python|template.*foo.*django.*hello.*world.*python|foo.*template.*django.*hello.*world.*python|django.*template.*foo.*hello.*world.*python|template.*django.*foo.*hello.*world.*python|hello.*django.*template.*foo.*world.*python|django.*hello.*template.*foo.*world.*python|template.*hello.*django.*foo.*world.*python|hello.*template.*django.*foo.*world.*python|django.*template.*hello.*foo.*world.*python|template.*django.*hello.*foo.*world.*python|hello.*foo.*template.*django.*world.*python|foo.*hello.*template.*django.*world.*python|template.*hello.*foo.*django.*world.*python|hello.*template.*foo.*django.*world.*python|foo.*template.*hello.*django.*world.*python|template.*foo.*hello.*django.*world.*python|hello.*foo.*django.*template.*world.*python|foo.*hello.*django.*template.*world.*python|django.*hello.*foo.*template.*world.*python|hello.*django.*foo.*template.*world.*python|foo.*django.*hello.*template.*world.*python|django.*foo.*hello.*template.*world.*python|world.*foo.*django.*template.*hello.*python|foo.*world.*django.*template.*hello.*python|django.*world.*foo.*template.*hello.*python|world.*django.*foo.*template.*hello.*python|foo.*django.*world.*template.*hello.*python|django.*foo.*world.*template.*hello.*python|template.*foo.*django.*world.*hello.*python|foo.*template.*django.*world.*hello.*python|django.*template.*foo.*world.*hello.*python|template.*django.*foo.*world.*hello.*python|foo.*django.*template.*world.*hello.*python|django.*foo.*template.*world.*hello.*python|template.*world.*django.*foo.*hello.*python|world.*template.*django.*foo.*hello.*python|django.*template.*world.*foo.*hello.*python|template.*django.*world.*foo.*hello.*python|world.*django.*template.*foo.*hello.*python|django.*world.*template.*foo.*hello.*python|template.*world.*foo.*django.*hello.*python|world.*template.*foo.*django.*hello.*python|foo.*template.*world.*django.*hello.*python|template.*foo.*world.*django.*hello.*python|world.*foo.*template.*django.*hello.*python|foo.*world.*template.*django.*hello.*python|hello.*world.*foo.*django.*template.*python|world.*hello.*foo.*django.*template.*python|foo.*hello.*world.*django.*template.*python|hello.*foo.*world.*django.*template.*python|world.*foo.*hello.*django.*template.*python|foo.*world.*hello.*django.*template.*python|django.*world.*foo.*hello.*template.*python|world.*django.*foo.*hello.*template.*python|foo.*django.*world.*hello.*template.*python|django.*foo.*world.*hello.*template.*python|world.*foo.*django.*hello.*template.*python|foo.*world.*django.*hello.*template.*python|django.*hello.*foo.*world.*template.*python|hello.*django.*foo.*world.*template.*python|foo.*django.*hello.*world.*template.*python|django.*foo.*hello.*world.*template.*python|hello.*foo.*django.*world.*template.*python|foo.*hello.*django.*world.*template.*python|django.*hello.*world.*foo.*template.*python|hello.*django.*world.*foo.*template.*python|world.*django.*hello.*foo.*template.*python|django.*world.*hello.*foo.*template.*python|hello.*world.*django.*foo.*template.*python|world.*hello.*django.*foo.*template.*python|template.*hello.*world.*foo.*django.*python|hello.*template.*world.*foo.*django.*python|world.*template.*hello.*foo.*django.*python|template.*world.*hello.*foo.*django.*python|hello.*world.*template.*foo.*django.*python|world.*hello.*template.*foo.*django.*python|foo.*hello.*world.*template.*django.*python|hello.*foo.*world.*template.*django.*python|world.*foo.*hello.*template.*django.*python|foo.*world.*hello.*template.*django.*python|hello.*world.*foo.*template.*django.*python|world.*hello.*foo.*template.*django.*python|foo.*template.*world.*hello.*django.*python|template.*foo.*world.*hello.*django.*python|world.*foo.*template.*hello.*django.*python|foo.*world.*template.*hello.*django.*python|template.*world.*foo.*hello.*django.*python|world.*template.*foo.*hello.*django.*python|foo.*template.*hello.*world.*django.*python|template.*foo.*hello.*world.*django.*python|hello.*foo.*template.*world.*django.*python|foo.*hello.*template.*world.*django.*python|template.*hello.*foo.*world.*django.*python|hello.*template.*foo.*world.*django.*python|django.*template.*hello.*world.*foo.*python|template.*django.*hello.*world.*foo.*python|hello.*django.*template.*world.*foo.*python|django.*hello.*template.*world.*foo.*python|template.*hello.*django.*world.*foo.*python|hello.*template.*django.*world.*foo.*python|world.*template.*hello.*django.*foo.*python|template.*world.*hello.*django.*foo.*python|hello.*world.*template.*django.*foo.*python|world.*hello.*template.*django.*foo.*python|template.*hello.*world.*django.*foo.*python|hello.*template.*world.*django.*foo.*python|world.*django.*hello.*template.*foo.*python|django.*world.*hello.*template.*foo.*python|hello.*world.*django.*template.*foo.*python|world.*hello.*django.*template.*foo.*python|django.*hello.*world.*template.*foo.*python|hello.*django.*world.*template.*foo.*python|world.*django.*template.*hello.*foo.*python|django.*world.*template.*hello.*foo.*python|template.*world.*django.*hello.*foo.*python|world.*template.*django.*hello.*foo.*python|django.*template.*world.*hello.*foo.*python|template.*django.*world.*hello.*foo.*python|foo.*python.*template.*hello.*world.*django|python.*foo.*template.*hello.*world.*django|template.*foo.*python.*hello.*world.*django|foo.*template.*python.*hello.*world.*django|python.*template.*foo.*hello.*world.*django|template.*python.*foo.*hello.*world.*django|hello.*python.*template.*foo.*world.*django|python.*hello.*template.*foo.*world.*django|template.*hello.*python.*foo.*world.*django|hello.*template.*python.*foo.*world.*django|python.*template.*hello.*foo.*world.*django|template.*python.*hello.*foo.*world.*django|hello.*foo.*template.*python.*world.*django|foo.*hello.*template.*python.*world.*django|template.*hello.*foo.*python.*world.*django|hello.*template.*foo.*python.*world.*django|foo.*template.*hello.*python.*world.*django|template.*foo.*hello.*python.*world.*django|hello.*foo.*python.*template.*world.*django|foo.*hello.*python.*template.*world.*django|python.*hello.*foo.*template.*world.*django|hello.*python.*foo.*template.*world.*django|foo.*python.*hello.*template.*world.*django|python.*foo.*hello.*template.*world.*django|world.*foo.*python.*template.*hello.*django|foo.*world.*python.*template.*hello.*django|python.*world.*foo.*template.*hello.*django|world.*python.*foo.*template.*hello.*django|foo.*python.*world.*template.*hello.*django|python.*foo.*world.*template.*hello.*django|template.*foo.*python.*world.*hello.*django|foo.*template.*python.*world.*hello.*django|python.*template.*foo.*world.*hello.*django|template.*python.*foo.*world.*hello.*django|foo.*python.*template.*world.*hello.*django|python.*foo.*template.*world.*hello.*django|template.*world.*python.*foo.*hello.*django|world.*template.*python.*foo.*hello.*django|python.*template.*world.*foo.*hello.*django|template.*python.*world.*foo.*hello.*django|world.*python.*template.*foo.*hello.*django|python.*world.*template.*foo.*hello.*django|template.*world.*foo.*python.*hello.*django|world.*template.*foo.*python.*hello.*django|foo.*template.*world.*python.*hello.*django|template.*foo.*world.*python.*hello.*django|world.*foo.*template.*python.*hello.*django|foo.*world.*template.*python.*hello.*django|hello.*world.*foo.*python.*template.*django|world.*hello.*foo.*python.*template.*django|foo.*hello.*world.*python.*template.*django|hello.*foo.*world.*python.*template.*django|world.*foo.*hello.*python.*template.*django|foo.*world.*hello.*python.*template.*django|python.*world.*foo.*hello.*template.*django|world.*python.*foo.*hello.*template.*django|foo.*python.*world.*hello.*template.*django|python.*foo.*world.*hello.*template.*django|world.*foo.*python.*hello.*template.*django|foo.*world.*python.*hello.*template.*django|python.*hello.*foo.*world.*template.*django|hello.*python.*foo.*world.*template.*django|foo.*python.*hello.*world.*template.*django|python.*foo.*hello.*world.*template.*django|hello.*foo.*python.*world.*template.*django|foo.*hello.*python.*world.*template.*django|python.*hello.*world.*foo.*template.*django|hello.*python.*world.*foo.*template.*django|world.*python.*hello.*foo.*template.*django|python.*world.*hello.*foo.*template.*django|hello.*world.*python.*foo.*template.*django|world.*hello.*python.*foo.*template.*django|template.*hello.*world.*foo.*python.*django|hello.*template.*world.*foo.*python.*django|world.*template.*hello.*foo.*python.*django|template.*world.*hello.*foo.*python.*django|hello.*world.*template.*foo.*python.*django|world.*hello.*template.*foo.*python.*django|foo.*hello.*world.*template.*python.*django|hello.*foo.*world.*template.*python.*django|world.*foo.*hello.*template.*python.*django|foo.*world.*hello.*template.*python.*django|hello.*world.*foo.*template.*python.*django|world.*hello.*foo.*template.*python.*django|foo.*template.*world.*hello.*python.*django|template.*foo.*world.*hello.*python.*django|world.*foo.*template.*hello.*python.*django|foo.*world.*template.*hello.*python.*django|template.*world.*foo.*hello.*python.*django|world.*template.*foo.*hello.*python.*django|foo.*template.*hello.*world.*python.*django|template.*foo.*hello.*world.*python.*django|hello.*foo.*template.*world.*python.*django|foo.*hello.*template.*world.*python.*django|template.*hello.*foo.*world.*python.*django|hello.*template.*foo.*world.*python.*django|python.*template.*hello.*world.*foo.*django|template.*python.*hello.*world.*foo.*django|hello.*python.*template.*world.*foo.*django|python.*hello.*template.*world.*foo.*django|template.*hello.*python.*world.*foo.*django|hello.*template.*python.*world.*foo.*django|world.*template.*hello.*python.*foo.*django|template.*world.*hello.*python.*foo.*django|hello.*world.*template.*python.*foo.*django|world.*hello.*template.*python.*foo.*django|template.*hello.*world.*python.*foo.*django|hello.*template.*world.*python.*foo.*django|world.*python.*hello.*template.*foo.*django|python.*world.*hello.*template.*foo.*django|hello.*world.*python.*template.*foo.*django|world.*hello.*python.*template.*foo.*django|python.*hello.*world.*template.*foo.*django|hello.*python.*world.*template.*foo.*django|world.*python.*template.*hello.*foo.*django|python.*world.*template.*hello.*foo.*django|template.*world.*python.*hello.*foo.*django|world.*template.*python.*hello.*foo.*django|python.*template.*world.*hello.*foo.*django|template.*python.*world.*hello.*foo.*django|foo.*python.*django.*hello.*world.*template|python.*foo.*django.*hello.*world.*template|django.*foo.*python.*hello.*world.*template|foo.*django.*python.*hello.*world.*template|python.*django.*foo.*hello.*world.*template|django.*python.*foo.*hello.*world.*template|hello.*python.*django.*foo.*world.*template|python.*hello.*django.*foo.*world.*template|django.*hello.*python.*foo.*world.*template|hello.*django.*python.*foo.*world.*template|python.*django.*hello.*foo.*world.*template|django.*python.*hello.*foo.*world.*template|hello.*foo.*django.*python.*world.*template|foo.*hello.*django.*python.*world.*template|django.*hello.*foo.*python.*world.*template|hello.*django.*foo.*python.*world.*template|foo.*django.*hello.*python.*world.*template|django.*foo.*hello.*python.*world.*template|hello.*foo.*python.*django.*world.*template|foo.*hello.*python.*django.*world.*template|python.*hello.*foo.*django.*world.*template|hello.*python.*foo.*django.*world.*template|foo.*python.*hello.*django.*world.*template|python.*foo.*hello.*django.*world.*template|world.*foo.*python.*django.*hello.*template|foo.*world.*python.*django.*hello.*template|python.*world.*foo.*django.*hello.*template|world.*python.*foo.*django.*hello.*template|foo.*python.*world.*django.*hello.*template|python.*foo.*world.*django.*hello.*template|django.*foo.*python.*world.*hello.*template|foo.*django.*python.*world.*hello.*template|python.*django.*foo.*world.*hello.*template|django.*python.*foo.*world.*hello.*template|foo.*python.*django.*world.*hello.*template|python.*foo.*django.*world.*hello.*template|django.*world.*python.*foo.*hello.*template|world.*django.*python.*foo.*hello.*template|python.*django.*world.*foo.*hello.*template|django.*python.*world.*foo.*hello.*template|world.*python.*django.*foo.*hello.*template|python.*world.*django.*foo.*hello.*template|django.*world.*foo.*python.*hello.*template|world.*django.*foo.*python.*hello.*template|foo.*django.*world.*python.*hello.*template|django.*foo.*world.*python.*hello.*template|world.*foo.*django.*python.*hello.*template|foo.*world.*django.*python.*hello.*template|hello.*world.*foo.*python.*django.*template|world.*hello.*foo.*python.*django.*template|foo.*hello.*world.*python.*django.*template|hello.*foo.*world.*python.*django.*template|world.*foo.*hello.*python.*django.*template|foo.*world.*hello.*python.*django.*template|python.*world.*foo.*hello.*django.*template|world.*python.*foo.*hello.*django.*template|foo.*python.*world.*hello.*django.*template|python.*foo.*world.*hello.*django.*template|world.*foo.*python.*hello.*django.*template|foo.*world.*python.*hello.*django.*template|python.*hello.*foo.*world.*django.*template|hello.*python.*foo.*world.*django.*template|foo.*python.*hello.*world.*django.*template|python.*foo.*hello.*world.*django.*template|hello.*foo.*python.*world.*django.*template|foo.*hello.*python.*world.*django.*template|python.*hello.*world.*foo.*django.*template|hello.*python.*world.*foo.*django.*template|world.*python.*hello.*foo.*django.*template|python.*world.*hello.*foo.*django.*template|hello.*world.*python.*foo.*django.*template|world.*hello.*python.*foo.*django.*template|django.*hello.*world.*foo.*python.*template|hello.*django.*world.*foo.*python.*template|world.*django.*hello.*foo.*python.*template|django.*world.*hello.*foo.*python.*template|hello.*world.*django.*foo.*python.*template|world.*hello.*django.*foo.*python.*template|foo.*hello.*world.*django.*python.*template|hello.*foo.*world.*django.*python.*template|world.*foo.*hello.*django.*python.*template|foo.*world.*hello.*django.*python.*template|hello.*world.*foo.*django.*python.*template|world.*hello.*foo.*django.*python.*template|foo.*django.*world.*hello.*python.*template|django.*foo.*world.*hello.*python.*template|world.*foo.*django.*hello.*python.*template|foo.*world.*django.*hello.*python.*template|django.*world.*foo.*hello.*python.*template|world.*django.*foo.*hello.*python.*template|foo.*django.*hello.*world.*python.*template|django.*foo.*hello.*world.*python.*template|hello.*foo.*django.*world.*python.*template|foo.*hello.*django.*world.*python.*template|django.*hello.*foo.*world.*python.*template|hello.*django.*foo.*world.*python.*template|python.*django.*hello.*world.*foo.*template|django.*python.*hello.*world.*foo.*template|hello.*python.*django.*world.*foo.*template|python.*hello.*django.*world.*foo.*template|django.*hello.*python.*world.*foo.*template|hello.*django.*python.*world.*foo.*template|world.*django.*hello.*python.*foo.*template|django.*world.*hello.*python.*foo.*template|hello.*world.*django.*python.*foo.*template|world.*hello.*django.*python.*foo.*template|django.*hello.*world.*python.*foo.*template|hello.*django.*world.*python.*foo.*template|world.*python.*hello.*django.*foo.*template|python.*world.*hello.*django.*foo.*template|hello.*world.*python.*django.*foo.*template|world.*hello.*python.*django.*foo.*template|python.*hello.*world.*django.*foo.*template|hello.*python.*world.*django.*foo.*template|world.*python.*django.*hello.*foo.*template|python.*world.*django.*hello.*foo.*template|django.*world.*python.*hello.*foo.*template|world.*django.*python.*hello.*foo.*template|python.*django.*world.*hello.*foo.*template|django.*python.*world.*hello.*foo.*template|foo.*python.*django.*template.*world.*hello|python.*foo.*django.*template.*world.*hello|django.*foo.*python.*template.*world.*hello|foo.*django.*python.*template.*world.*hello|python.*django.*foo.*template.*world.*hello|django.*python.*foo.*template.*world.*hello|template.*python.*django.*foo.*world.*hello|python.*template.*django.*foo.*world.*hello|django.*template.*python.*foo.*world.*hello|template.*django.*python.*foo.*world.*hello|python.*django.*template.*foo.*world.*hello|django.*python.*template.*foo.*world.*hello|template.*foo.*django.*python.*world.*hello|foo.*template.*django.*python.*world.*hello|django.*template.*foo.*python.*world.*hello|template.*django.*foo.*python.*world.*hello|foo.*django.*template.*python.*world.*hello|django.*foo.*template.*python.*world.*hello|template.*foo.*python.*django.*world.*hello|foo.*template.*python.*django.*world.*hello|python.*template.*foo.*django.*world.*hello|template.*python.*foo.*django.*world.*hello|foo.*python.*template.*django.*world.*hello|python.*foo.*template.*django.*world.*hello|world.*foo.*python.*django.*template.*hello|foo.*world.*python.*django.*template.*hello|python.*world.*foo.*django.*template.*hello|world.*python.*foo.*django.*template.*hello|foo.*python.*world.*django.*template.*hello|python.*foo.*world.*django.*template.*hello|django.*foo.*python.*world.*template.*hello|foo.*django.*python.*world.*template.*hello|python.*django.*foo.*world.*template.*hello|django.*python.*foo.*world.*template.*hello|foo.*python.*django.*world.*template.*hello|python.*foo.*django.*world.*template.*hello|django.*world.*python.*foo.*template.*hello|world.*django.*python.*foo.*template.*hello|python.*django.*world.*foo.*template.*hello|django.*python.*world.*foo.*template.*hello|world.*python.*django.*foo.*template.*hello|python.*world.*django.*foo.*template.*hello|django.*world.*foo.*python.*template.*hello|world.*django.*foo.*python.*template.*hello|foo.*django.*world.*python.*template.*hello|django.*foo.*world.*python.*template.*hello|world.*foo.*django.*python.*template.*hello|foo.*world.*django.*python.*template.*hello|template.*world.*foo.*python.*django.*hello|world.*template.*foo.*python.*django.*hello|foo.*template.*world.*python.*django.*hello|template.*foo.*world.*python.*django.*hello|world.*foo.*template.*python.*django.*hello|foo.*world.*template.*python.*django.*hello|python.*world.*foo.*template.*django.*hello|world.*python.*foo.*template.*django.*hello|foo.*python.*world.*template.*django.*hello|python.*foo.*world.*template.*django.*hello|world.*foo.*python.*template.*django.*hello|foo.*world.*python.*template.*django.*hello|python.*template.*foo.*world.*django.*hello|template.*python.*foo.*world.*django.*hello|foo.*python.*template.*world.*django.*hello|python.*foo.*template.*world.*django.*hello|template.*foo.*python.*world.*django.*hello|foo.*template.*python.*world.*django.*hello|python.*template.*world.*foo.*django.*hello|template.*python.*world.*foo.*django.*hello|world.*python.*template.*foo.*django.*hello|python.*world.*template.*foo.*django.*hello|template.*world.*python.*foo.*django.*hello|world.*template.*python.*foo.*django.*hello|django.*template.*world.*foo.*python.*hello|template.*django.*world.*foo.*python.*hello|world.*django.*template.*foo.*python.*hello|django.*world.*template.*foo.*python.*hello|template.*world.*django.*foo.*python.*hello|world.*template.*django.*foo.*python.*hello|foo.*template.*world.*django.*python.*hello|template.*foo.*world.*django.*python.*hello|world.*foo.*template.*django.*python.*hello|foo.*world.*template.*django.*python.*hello|template.*world.*foo.*django.*python.*hello|world.*template.*foo.*django.*python.*hello|foo.*django.*world.*template.*python.*hello|django.*foo.*world.*template.*python.*hello|world.*foo.*django.*template.*python.*hello|foo.*world.*django.*template.*python.*hello|django.*world.*foo.*template.*python.*hello|world.*django.*foo.*template.*python.*hello|foo.*django.*template.*world.*python.*hello|django.*foo.*template.*world.*python.*hello|template.*foo.*django.*world.*python.*hello|foo.*template.*django.*world.*python.*hello|django.*template.*foo.*world.*python.*hello|template.*django.*foo.*world.*python.*hello|python.*django.*template.*world.*foo.*hello|django.*python.*template.*world.*foo.*hello|template.*python.*django.*world.*foo.*hello|python.*template.*django.*world.*foo.*hello|django.*template.*python.*world.*foo.*hello|template.*django.*python.*world.*foo.*hello|world.*django.*template.*python.*foo.*hello|django.*world.*template.*python.*foo.*hello|template.*world.*django.*python.*foo.*hello|world.*template.*django.*python.*foo.*hello|django.*template.*world.*python.*foo.*hello|template.*django.*world.*python.*foo.*hello|world.*python.*template.*django.*foo.*hello|python.*world.*template.*django.*foo.*hello|template.*world.*python.*django.*foo.*hello|world.*template.*python.*django.*foo.*hello|python.*template.*world.*django.*foo.*hello|template.*python.*world.*django.*foo.*hello|world.*python.*django.*template.*foo.*hello|python.*world.*django.*template.*foo.*hello|django.*world.*python.*template.*foo.*hello|world.*django.*python.*template.*foo.*hello|python.*django.*world.*template.*foo.*hello|django.*python.*world.*template.*foo.*hello|foo.*python.*django.*template.*hello.*world|python.*foo.*django.*template.*hello.*world|django.*foo.*python.*template.*hello.*world|foo.*django.*python.*template.*hello.*world|python.*django.*foo.*template.*hello.*world|django.*python.*foo.*template.*hello.*world|template.*python.*django.*foo.*hello.*world|python.*template.*django.*foo.*hello.*world|django.*template.*python.*foo.*hello.*world|template.*django.*python.*foo.*hello.*world|python.*django.*template.*foo.*hello.*world|django.*python.*template.*foo.*hello.*world|template.*foo.*django.*python.*hello.*world|foo.*template.*django.*python.*hello.*world|django.*template.*foo.*python.*hello.*world|template.*django.*foo.*python.*hello.*world|foo.*django.*template.*python.*hello.*world|django.*foo.*template.*python.*hello.*world|template.*foo.*python.*django.*hello.*world|foo.*template.*python.*django.*hello.*world|python.*template.*foo.*django.*hello.*world|template.*python.*foo.*django.*hello.*world|foo.*python.*template.*django.*hello.*world|python.*foo.*template.*django.*hello.*world|hello.*foo.*python.*django.*template.*world|foo.*hello.*python.*django.*template.*world|python.*hello.*foo.*django.*template.*world|hello.*python.*foo.*django.*template.*world|foo.*python.*hello.*django.*template.*world|python.*foo.*hello.*django.*template.*world|django.*foo.*python.*hello.*template.*world|foo.*django.*python.*hello.*template.*world|python.*django.*foo.*hello.*template.*world|django.*python.*foo.*hello.*template.*world|foo.*python.*django.*hello.*template.*world|python.*foo.*django.*hello.*template.*world|django.*hello.*python.*foo.*template.*world|hello.*django.*python.*foo.*template.*world|python.*django.*hello.*foo.*template.*world|django.*python.*hello.*foo.*template.*world|hello.*python.*django.*foo.*template.*world|python.*hello.*django.*foo.*template.*world|django.*hello.*foo.*python.*template.*world|hello.*django.*foo.*python.*template.*world|foo.*django.*hello.*python.*template.*world|django.*foo.*hello.*python.*template.*world|hello.*foo.*django.*python.*template.*world|foo.*hello.*django.*python.*template.*world|template.*hello.*foo.*python.*django.*world|hello.*template.*foo.*python.*django.*world|foo.*template.*hello.*python.*django.*world|template.*foo.*hello.*python.*django.*world|hello.*foo.*template.*python.*django.*world|foo.*hello.*template.*python.*django.*world|python.*hello.*foo.*template.*django.*world|hello.*python.*foo.*template.*django.*world|foo.*python.*hello.*template.*django.*world|python.*foo.*hello.*template.*django.*world|hello.*foo.*python.*template.*django.*world|foo.*hello.*python.*template.*django.*world|python.*template.*foo.*hello.*django.*world|template.*python.*foo.*hello.*django.*world|foo.*python.*template.*hello.*django.*world|python.*foo.*template.*hello.*django.*world|template.*foo.*python.*hello.*django.*world|foo.*template.*python.*hello.*django.*world|python.*template.*hello.*foo.*django.*world|template.*python.*hello.*foo.*django.*world|hello.*python.*template.*foo.*django.*world|python.*hello.*template.*foo.*django.*world|template.*hello.*python.*foo.*django.*world|hello.*template.*python.*foo.*django.*world|django.*template.*hello.*foo.*python.*world|template.*django.*hello.*foo.*python.*world|hello.*django.*template.*foo.*python.*world|django.*hello.*template.*foo.*python.*world|template.*hello.*django.*foo.*python.*world|hello.*template.*django.*foo.*python.*world|foo.*template.*hello.*django.*python.*world|template.*foo.*hello.*django.*python.*world|hello.*foo.*template.*django.*python.*world|foo.*hello.*template.*django.*python.*world|template.*hello.*foo.*django.*python.*world|hello.*template.*foo.*django.*python.*world|foo.*django.*hello.*template.*python.*world|django.*foo.*hello.*template.*python.*world|hello.*foo.*django.*template.*python.*world|foo.*hello.*django.*template.*python.*world|django.*hello.*foo.*template.*python.*world|hello.*django.*foo.*template.*python.*world|foo.*django.*template.*hello.*python.*world|django.*foo.*template.*hello.*python.*world|template.*foo.*django.*hello.*python.*world|foo.*template.*django.*hello.*python.*world|django.*template.*foo.*hello.*python.*world|template.*django.*foo.*hello.*python.*world|python.*django.*template.*hello.*foo.*world|django.*python.*template.*hello.*foo.*world|template.*python.*django.*hello.*foo.*world|python.*template.*django.*hello.*foo.*world|django.*template.*python.*hello.*foo.*world|template.*django.*python.*hello.*foo.*world|hello.*django.*template.*python.*foo.*world|django.*hello.*template.*python.*foo.*world|template.*hello.*django.*python.*foo.*world|hello.*template.*django.*python.*foo.*world|django.*template.*hello.*python.*foo.*world|template.*django.*hello.*python.*foo.*world|hello.*python.*template.*django.*foo.*world|python.*hello.*template.*django.*foo.*world|template.*hello.*python.*django.*foo.*world|hello.*template.*python.*django.*foo.*world|python.*template.*hello.*django.*foo.*world|template.*python.*hello.*django.*foo.*world|hello.*python.*django.*template.*foo.*world|python.*hello.*django.*template.*foo.*world|django.*hello.*python.*template.*foo.*world|hello.*django.*python.*template.*foo.*world|python.*django.*hello.*template.*foo.*world|django.*python.*hello.*template.*foo.*world/?regexp=true&limit=200&offset=0&with_visit=true HTTP/1.1" 200 2
"GET /browse/search/?q=python%20django%20template%20hello%20world%20foo%20test&with_visit&with_content HTTP/1.1" 200 5759
"GET /api/1/resolve/python%20django%20template%20hello%20world%20foo%20test/ HTTP/1.1" 400 474
code 414, message Request-URI Too Long

Nevertheless, we should still issue a search request even when the number of keywords is high.
In that case, we do not use the permutation algorithm and simply pass the raw input string
as search parameter. So this diff needs more work.

By the way, can you issue the following command in your shell:

arc set-config history.immutable true

It will prevent arcanist to overwrite your commit message when you submit a diff.
See what I get when I issue git log on your feature branch:

✔ ~/swh/swh-environment/swh-web [arcpatch-D1295 L|⚑ 59] 
19:40 $ git log
commit c3723cb92c8fd14093d6a4dfe4961443b0b9e7ed
Author: kalpitk <kkothari1999@gmail.com>
Date:   Tue Mar 26 19:40:25 2019 +0100

    prevent high memory usage
    
    Summary: Heap permutation for more than 8 strings takes high amount of memory
    
    Reviewers: #reviewers
    
    Subscribers: swh-public-ci
    
    Differential Revision: https://forge.softwareheritage.org/D1295

There is a lot of arcanist generated noise we do not want to have in the history log.

Also, can you address my last comment in D1290 ? Once it is done, I will be able to land that diff.

This revision now requires changes to proceed.Mar 26 2019, 7:49 PM
zack requested changes to this revision.Mar 26 2019, 7:50 PM
zack added a subscriber: zack.
zack added inline comments.
swh/web/assets/src/bundles/browse/origin-search.js
100–109

I've no opinion on the merits of this proposed change (@anlambert will probably comment on that), but *if* this is to go in , this line definitely needs a comment explaining why the magic number and what it does.

  • add case for bigger array lengths

@anlambert given we have a trigram index on origin URLs, have you ever tried to use the various similarity operators document at https://www.postgresql.org/docs/11/pgtrgm.html instead of generating all possible permutations for regexs?
I'm assuming (probably too naively) that you can just do a big select on the URLs, sorting by similarity and possibly filtering on a threshold to return meaningful results. But it's not like I've actually tested it…

(yes, this is orthogonal to this proposed change, but can make its need moot if it works)

In D1295#27648, @zack wrote:

@anlambert given we have a trigram index on origin URLs, have you ever tried to use the various similarity operators document at https://www.postgresql.org/docs/11/pgtrgm.html instead of generating all possible permutations for regexs?
I'm assuming (probably too naively) that you can just do a big select on the URLs, sorting by similarity and possibly filtering on a threshold to return meaningful results. But it's not like I've actually tested it…

or, actually, we can just also add a fulltext index to URLs and be done with it https://www.postgresql.org/docs/11/textsearch-intro.html#TEXTSEARCH-MATCHING

In D1295#27649, @zack wrote:

or, actually, we can just also add a fulltext index to URLs and be done with it https://www.postgresql.org/docs/11/textsearch-intro.html#TEXTSEARCH-MATCHING

As just pointed out by @olasd on IRC, this is in fact D561. Fancy taking that up, @kalpitk ?

In D1295#27652, @zack wrote:
In D1295#27649, @zack wrote:

or, actually, we can just also add a fulltext index to URLs and be done with it https://www.postgresql.org/docs/11/textsearch-intro.html#TEXTSEARCH-MATCHING

As just pointed out by @olasd on IRC, this is in fact D561. Fancy taking that up, @kalpitk ?

I can try, but it may take some time. I'll see all the possible options.

Accepting this until a better origin search machinery (D561) gets implemented and tested.

This revision is now accepted and ready to land.Mar 28 2019, 9:52 PM
zack requested changes to this revision.Mar 28 2019, 9:53 PM

Oups, sorry, didn't mean to accept this, only to remove myself from reviewers.
I'll let @anlambert finish the actual review.

This revision now requires changes to proceed.Mar 28 2019, 9:53 PM

@kalpitk , I have already landed this changes. You can close this diff now.

This revision is now accepted and ready to land.Mar 29 2019, 11:22 AM