Page MenuHomeSoftware Heritage

investigate slow uffizi read performances from the object store
Closed, MigratedEdits Locked

Description

In the context of T7 we started copying uffizi → banco. The estimated ETA was ~50 days for ~50 TB, i.e., 1TB/day which is definitely too slow.

Oddly enough, the bottleneck seems to be neither the network, nor the writes on banco, but rather the reads (!) on uffizi.

We need to figure out why it is so.

Event Timeline

Here are some bonnie++ tests on both uffizi and banco. They seem consistent with the fact that reads on uffizi from the object storage are much slower (factor 3x) when compared with banco. But further investigation is needed.

zack@banco$ bonnie++ -d /srv/softwareheritage/objects/01/bonnie++ 
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
banco        96584M  1211  99 952310  98 569833  66  2612  99 1503015  88 576.2  22
Latency              8600us   17858us     325ms    7576us     336ms     133ms
Version  1.97       ------Sequential Create------ --------Random Create--------
banco               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 32351  81 +++++ +++ +++++ +++ 32666  82 +++++ +++ +++++ +++
Latency             22834us     136us   11007us   14554us      12us   16350us
1.97,1.97,banco,1,1453364369,96584M,,1211,99,952310,98,569833,66,2612,99,1503015,88,576.2,22,16,,,,,32351,81,+++++,+++,+++++,+++,32666,82,+++++,+++,+++++,+++,8600us,17858us,325ms,7576us,336ms,133ms,22834us,136us,11007us,14554us,12us,16350us
zack@banco$ bonnie++ -d /srv/softwareheritage/objects/a1/bonnie++ 
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
banco        96584M  1230  99 988938  98 558315  67  2561  97 1566189  88 586.5  22
Latency              7934us   15883us     346ms   21770us     140ms   98680us
Version  1.97       ------Sequential Create------ --------Random Create--------
banco               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              6700us     131us   10769us     301us      15us     104us
1.97,1.97,banco,1,1453363389,96584M,,1230,99,988938,98,558315,67,2561,97,1566189,88,586.5,22,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,7934us,15883us,346ms,21770us,140ms,98680us,6700us,131us,10769us,301us,15us,104us
zack@uffizi$ bonnie++ -d /srv/softwareheritage/objects/01/bonnie++
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
uffizi          95G  1378  98 755458  98 364821  58  2908  95 694353  42 257.7  36
Latency              6232us   15146us     390ms   50966us     338ms     218ms
Version  1.97       ------Sequential Create------ --------Random Create--------
uffizi              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22208  79 +++++ +++ 30949  88 23446  81 +++++ +++ 29881  88
Latency             33782us     130us     195us   22324us      30us     194us
1.97,1.97,uffizi,1,1453420976,95G,,1378,98,755458,98,364821,58,2908,95,694353,42,257.7,36,16,,,,,22208,79,+++++,+++,30949,88,23446,81,+++++,+++,29881,88,6232us,15146us,390ms,50966us,338ms,218ms,33782us,130us,195us,22324us,30us,194us
zack@uffizi$ bonnie++ -d /srv/softwareheritage/objects/a1/bonnie++
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
uffizi          95G  1351  98 686191  98 327914  54  3006  97 599862  45 204.9  44
Latency              9124us   13713us     302ms   17296us     329ms     248ms
Version  1.97       ------Sequential Create------ --------Random Create--------
uffizi              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22364  80 +++++ +++ 29042  88 21705  83 +++++ +++ 25380  89
Latency             27162us     210us     224us   19690us      65us     337us
1.97,1.97,uffizi,1,1453421431,95G,,1351,98,686191,98,327914,54,3006,97,599862,45,204.9,44,16,,,,,22364,80,+++++,+++,29042,88,21705,83,+++++,+++,25380,89,9124us,13713us,302ms,17296us,329ms,248ms,27162us,210us,224us,19690us,65us,337us
zack removed zack as the assignee of this task.Jan 21 2016, 10:24 PM

The main bottleneck turned out to be seek time, that for 1.6B files really adds up.

There remain inconsistency among the performance of the different storages, within and outside libvirt. Separate tasks to investigate/fix those will be filed separately.

olasd changed the visibility from "All Users" to "Public (No Login Required)".May 13 2016, 5:08 PM