Page MenuHomeSoftware Heritage

[cassandra] directory and content read benchmarks
Started, Work in Progress, NormalPublic

Description

Top issue around directory_ls and content retrieval performance issue

Will be used to track the tests of D6228, D6229 and potential others

Event Timeline

vsellier changed the task status from Open to Work in Progress.Wed, Sep 15, 11:11 AM
vsellier triaged this task as Normal priority.
vsellier created this task.
vsellier renamed this task from [cassandra] directory and content read benchmarkss to [cassandra] directory and content read benchmarks.EditedWed, Sep 15, 11:19 AM
vsellier updated the task description. (Show Details)

The directory_ls and indirectly the get_content performace was tested with this small script: P1163
A cold restart (all buffer cleared, cassandra restarted) is done between each tests (P1164)

This is the results of the different runs:

  • run a directory_ls on a big directory
  • postgresql (saam)
vsellier@saam ~ % python3 directory_ls.py c864e846cb339a94da9fd91ae12cabcf083a8685 
c864e846cb339a94da9fd91ae12cabcf083a8685: 8.9192s
  • cassandra

10 directory_ls was launch to test the impacts of having the data in cache:

run idone-by-one[1]concurrent [2]
1233.6952s184.1209s
2102.1767s89.1716s
341.1162s33.0051s
425.4258s21.7695s
518.2770s19.0004s
616.8347s14.6394s
715.4968s13.4750s
814.1320s12.1201s
913.3825s10.6304s
1013.1336s11.1261s

[1] D6228 and D6229 reverted
[2] master of swh-storage https://archive.softwareheritage.org/swh:1:snp:dd585ae36b25c37fc9a4b5ab16fb4d0482a075a7;origin=https://forge.softwareheritage.org/source/swh-storage.git

The concurrent version is globally faster but the performances are not crazy compared to the postgresql version

  • run 2000 directory_ls with 50 threads in parallel

IN PROGRESS

time cat directory_2000.lst | parallel --bar -j50 python3 directory_ls.py {} | tee directory_ls_[concurrent|one-by-one]-2000.output
  • one-by-one:
real    11m34.692s
user    19m6.793s
sys     2m12.372s

The time is mostly the loading time of the longest repository:

bc0a1450e393f47cb34a6f1f1a6ee9206c739579: 670.7532s
  • concurrent :
real    9m4.044s
user    20m12.418s
sys     2m13.689s
bc0a1450e393f47cb34a6f1f1a6ee9206c739579: 520.5311s

2 flame graphs of the previous directory_ls:

  • one-by-one

first run (cache cold):

last run (cache hot):

  • concurrent:

first run (cache cold):

last run (cache hot):

Test of a the new D6269 patch:

  • directory_ls on c864e846cb339a94da9fd91ae12cabcf083a8685
runduration
123.0080s
210.0942s
311.0577s
47.4534s
57.1187s
67.2937s
76.4423s
86.3384s
96.2031s
106.0875s
  • directory_ls on bc0a1450e393f47cb34a6f1f1a6ee9206c739579 (the biggest directory in the 2000 ids)
runduration
167.8805s
238.4308s
333.2637s
430.7730s
530.1840s
627.8511s
728.2692s
827.8125s
927.4106s
1027.6957s

and the associated flamegraphes:

first run:

last run:

  • 2000 directory_ls with 50 thread in parallel:
real    1m40.011s
user    22m5.676s
sys     2m22.493s
bc0a1450e393f47cb34a6f1f1a6ee9206c739579: 70.3515s

much better than previously