investigate libvirt I/O slowdown
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	zack
	Jan 24 2016, 10:32 PM

Description

There seems to be a 50% read speedup (1.5x) when accessing uffizi's disk array storage directly from louvre (bare metal server) w.r.t. accessing the same storage via uffizi (libvirt/kvm guest on louvre). The speed up is there for writes too, but is less significant.
We need to find out whether that's "normal" and, if not, how to speed things up.

bonnie++ benchmarks for libvirt/kvm access (on a couple of shards):

zack@uffizi$ bonnie++ -d /srv/softwareheritage/objects/01/bonnie++

Seq Out			Seq In		Random
Char	Block	Rewrite	Char	Block	Seeks
K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	/s - CPU
1378 - 98%	755458 - 98%	364821 - 58%	2908 - 95%	694353 - 42%	257.7 - 36%
6232us	15146us	390ms	50966us	338ms	218ms
Seq Create			Random Create
Create	Read	Delete	Create	Read	Delete
/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU
22208 - 79%	+++++ - +++%	30949 - 88%	23446 - 81%	+++++ - +++%	29881 - 88%
33782us	130us	195us	22324us	30us	194us

zack@uffizi$ bonnie++ -d /srv/softwareheritage/objects/a1/bonnie++

Seq Out			Seq In		Random
Char	Block	Rewrite	Char	Block	Seeks
K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	/s - CPU
1351 - 98%	686191 - 98%	327914 - 54%	3006 - 97%	599862 - 45%	204.9 - 44%
9124us	13713us	302ms	17296us	329ms	248ms
Seq Create			Random Create
Create	Read	Delete	Create	Read	Delete
/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU
22364 - 80%	+++++ - +++%	29042 - 88%	21705 - 83%	+++++ - +++%	25380 - 89%
27162us	210us	224us	19690us	65us	337us

Same benchmarks accessing the same shards via louvre (bare metal):

zack@louvre$ mount|grep /mnt
/dev/mapper/vg--data-0 on /mnt type xfs (rw,relatime,attr2,inode64,noquota)
zack@louvre$ bonnie++ -d /mnt/bonnie

Seq Out			Seq In		Random
Char	Block	Rewrite	Char	Block	Seeks
K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	/s - CPU
1404 - 99%	633900 - 99%	478570 - 75%	2820 - 99%	919112 - 91%	345.1 - 27%
7770us	244ms	359ms	10967us	513ms	111ms
Seq Create			Random Create
Create	Read	Delete	Create	Read	Delete
/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU
28184 - 88%	+++++ - +++%	+++++ - +++%	28916 - 84%	+++++ - +++%	+++++ - +++%
478us	151us	14701us	444us	15us	43015us

zack@louvre$ mount | grep /mnt
/dev/mapper/vg--data-a on /mnt type xfs (rw,relatime,attr2,inode64,noquota)
zack@louvre$ bonnie++ -d /mnt/bonnie

Seq Out			Seq In		Random
Char	Block	Rewrite	Char	Block	Seeks
K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	/s - CPU
1317 - 99%	655989 - 99%	480398 - 75%	2571 - 95%	948253 - 92%	350.6 - 25%
9479us	297ms	10843ms	38326us	358ms	136ms
Seq Create			Random Create
Create	Read	Delete	Create	Read	Delete
/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU
28950 - 85%	+++++ - +++%	+++++ - +++%	29147 - 88%	+++++ - +++%	+++++ - +++%
478us	112us	13591us	355us	30us	24317us

Related Objects

Mentioned In: T284: investigate XFS tuning slowdown on uffizi w.r.t. banco

Event Timeline

zack created this task.Jan 24 2016, 10:32 PM

Herald added a project: Staff. · View Herald TranscriptJan 24 2016, 10:32 PM

For completeness, here are the slowdown benchmarks for prado's SSD disks (bottom line: the slow down seems to be present there too, but "only" of the order of 20% or so):

zack@prado$ -d /srv/softwareheritage/postgres/bonnie

Seq Out			Seq In		Random
Char	Block	Rewrite	Char	Block	Seeks
K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	/s - CPU
572 - 99%	414675 - 93%	252410 - 48%	3080 - 99%	481180 - 39%	+++++ - +++%
21403us	180ms	1119ms	5057us	38838us	17914us
Seq Create			Random Create
Create	Read	Delete	Create	Read	Delete
/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU
32148 - 60%	+++++ - +++%	+++++ - +++%	+++++ - +++%	+++++ - +++%	+++++ - +++%
492us	499us	2279us	1073us	32us	2463us

zack@louvre$ mount | grep /mnt      
/dev/mapper/ssd-part1 on /mnt type ext4 (rw,relatime,stripe=64,data=ordered)
zack@louvre$ bonnie++ -d /mnt/bonnie

Seq Out			Seq In		Random
Char	Block	Rewrite	Char	Block	Seeks
K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	K/s - CPU	/s - CPU
607 - 99%	404439 - 84%	334773 - 58%	3224 - 99%	588338 - 50%	+++++ - +++%
17549us	141ms	231ms	9394us	41663us	65093us
Seq Create			Random Create
Create	Read	Delete	Create	Read	Delete
/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU	/sec - CPU
+++++ - +++%	+++++ - +++%	+++++ - +++%	+++++ - +++%	+++++ - +++%	+++++ - +++%
6331us	653us	916us	391us	24us	1106us

Full benchmark data are available at: https://intranet.softwareheritage.org/index.php?title=User:StefanoZacchiroli/Disk_array_benchmark

zack mentioned this in T284: investigate XFS tuning slowdown on uffizi w.r.t. banco.Jan 24 2016, 10:43 PM

zack renamed this task from investigate libvirt I/O slow down to investigate libvirt I/O slowdown.Jan 24 2016, 10:45 PM

zack removed projects: System administrators, Staff.Mar 10 2016, 5:51 PM

zack added a project: System administration.Apr 1 2016, 10:20 AM

olasd changed the visibility from "All Users" to "Public (No Login Required)".May 13 2016, 5:08 PM

@douardda: can I punt this to you to either further investigate or just close as Invalid? 3 years later it might no longer be relevant…

Closing due to inactivity. Feel free to reopen if needed.

This task has been migrated to GitLab.

investigate libvirt I/O slowdownClosed, MigratedEdits LockedActions

Description

Related Objects

Event Timeline

investigate libvirt I/O slowdown
Closed, MigratedEdits Locked
Actions