This should make it run up to 100 times faster, even on average directories.
Depends on D6228.
Details
Details
- Reviewers
olasd - Group Reviewers
Reviewers - Commits
- rDSTO5facf661bf70: cassandra: Make directory_ls fetch contents in batch instead of one-by-one
Diff Detail
Diff Detail
- Repository
- rDSTO Storage manager
- Lint
Automatic diff as part of commit; lint not applicable. - Unit
Automatic diff as part of commit; unit tests not applicable.
Event Timeline
Comment Actions
Build is green
Patch application report for D6229 (id=22540)
Could not rebase; Attempt merge onto 50fb54f212...
Updating 50fb54f2..5facf661 Fast-forward swh/storage/cassandra/cql.py | 20 +++++---- swh/storage/cassandra/storage.py | 81 ++++++++++++++++++++++++------------- swh/storage/in_memory.py | 7 ++-- swh/storage/tests/test_cassandra.py | 12 +++--- 4 files changed, 75 insertions(+), 45 deletions(-)
Changes applied before test
commit 5facf661bf70973bde09367706141990de77638d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Thu Sep 9 15:45:30 2021 +0200
cassandra: Make directory_ls fetch contents in batch instead of one-by-one
This should make it run up to 100 times faster, even on average directories.
commit 0570a426efa5c89f85b407b5c3c53d6e9615cd06
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Thu Sep 9 15:30:09 2021 +0200
content_get: Fetch rows concurrently
Instead of fetching them one-by-one, with the very high latency this entails.
This is preliminary work to make `directory_ls` less painfully slow.See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1401/ for more details.