Pro virtualizaci na Linuxu už delší dobu používám výbornou platformu založenou na debianu – ProxmoxVE. Ačkoliv jsem v Proxmoxem velice spokojen, některé věci mi v něm vadí a tak jsem chtěl vyzkoušet postavit platformu vlastní nad Ubuntu, které mám daleko raději. Před stavbou samotnou jsem chtěl udělat test, jak je na tom Ubuntu a libvirt s výkonem IO operací.
Aby to bylo trochu zajímavější, chtěl jsem otestovat jak diskové obrazy qcow2, tak i raw. Qcow2 image byl před testem „nafouknut“ pomocí dd, abychom testovali reálnou výkonnost, ne rychlost alokace místa pro image. VPS image byly umístěny na LVM2 nad RAID1 polem.
Test jsem po prvních výsledcíh z Ubuntu přerušil a nechtělo se mi pokračovat, dám tedy k dispozici alespoň výledky testů qcow2 vs raw. Je mi jasné že tohle nejsou moc relevantní testy, ale pro mé účely posloužily a pár závěrů se z nich vyvodit dalo:
- Obrazy qcow2 jsou jednoznačně rychlejší než raw obrazy. Přisuzuji to nějaké cache, kterou qcow2 obrazy musejí mít.
- Tohle neplatí pro Ubuntu s libvirt, tam to vyjde skoro nastejno.
- Na Ubuntu s libvirt jsou qcow2 obrazy oproti Proxmoxu pomalejší.
Pro test byl použit následující stroj
- Dell PowerEdge 1950 G4V183J
- 2x Quad Core Xeon E5335 @ 2 Ghz
- 24GB DDR2 FB-DIMM 667 MHz
- LSI SAS1068 PCI-X Fusion-MPT SAS
- 2x FUJITSU MAX3073RC 15000rpm 73GB
Nafouknutí qcow2 obrazu
test1# dd if=/dev/urandom of=temp bs=10485760 count=2560
dd: writing `temp': No space left on device
1909+0 records in
1908+0 records out
20010160128 bytes (20 GB) copied, 2141.45 s, 9.3 MB/s
Informace o qcow2 obrazu
proxmox# qemu-img info test1.qcow2
image: test1.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 20G
cluster_size: 65536
ProxmoxVE balíky
- bonnie++ 1.96
- lvm2 2.02.95-1pve2
- pve-kernel-2.6.32-16-pve 2.6.32-80
- pve-qemu-kvm 1.2-7
- qemu-server 2.0-64
Konfigurace VPS pro test qcow2
-smp sockets=1,cores=4
-m 4096
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200
-drive file=test1.qcow2,if=none,id=drive-virtio0,aio=native
-device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100
Konfigurace VPS pro test raw
-smp sockets=1,cores=4
-m 4096
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200
-drive file=test2.raw,if=none,id=drive-virtio0,aio=native,cache=none
-device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100
Test pomocí hdparm na qcow2
test1# hdparm -tT /dev/vda
/dev/vda:
Timing cached reads: 6028 MB in 2.00 seconds = 3015.90 MB/sec
Timing buffered disk reads: 994 MB in 3.00 seconds = 331.18 MB/sec
test1# hdparm -tT /dev/vda
/dev/vda:
Timing cached reads: 5878 MB in 2.00 seconds = 2941.77 MB/sec
Timing buffered disk reads: 2022 MB in 3.00 seconds = 673.72 MB/sec
test1# hdparm -tT /dev/vda
/dev/vda:
Timing cached reads: 5574 MB in 2.00 seconds = 2789.37 MB/sec
Timing buffered disk reads: 2000 MB in 3.00 seconds = 666.18 MB/sec
Test pomocí hdparm na raw
test2# hdparm -tT /dev/vda
/dev/vda:
Timing cached reads: 5978 MB in 2.00 seconds = 2991.25 MB/sec
Timing buffered disk reads: 314 MB in 3.06 seconds = 102.48 MB/sec
test2# hdparm -tT /dev/vda
/dev/vda:
Timing cached reads: 5622 MB in 2.00 seconds = 2813.29 MB/sec
Timing buffered disk reads: 316 MB in 3.00 seconds = 105.23 MB/sec
test2# hdparm -tT /dev/vda
/dev/vda:
Timing cached reads: 5498 MB in 2.00 seconds = 2750.66 MB/sec
Timing buffered disk reads: 320 MB in 3.02 seconds = 105.99 MB/sec
Test pomocí bonnie++ na qcow2
test1# bonnie -d bonnie -u 0:0 -n 256
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
test1 8G 479 98 39390 6 34188 7 2110 97 885118 85 1666 33
Latency 19228us 57581ms 33761ms 13406us 4360us 60788us
Version 1.96 ------Sequential Create------ --------Random Create--------
test1 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
256 31812 69 355396 99 2622 4 17141 36 451849 99 1706 3
Latency 318ms 640us 11725ms 7707ms 114us 7031ms
1.96,1.96,test1,1,1356636922,8G,,479,98,39390,6,34188,7,2110,97,885118,85,1666,33,256,,,,,31812,69,355396,99,2622,4,17141,36,451849,99,1706,3,19228us,57581ms,33761ms,13406us,4360us,60788us,318ms,640us,11725ms,7707ms,114us,7031ms
Test pomocí bonnie++ na raw
test2# bonnie -d bonnie -u 0:0 -n 256
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
test2 8G 477 98 39592 10 24128 4 2083 96 99561 9 606.1 17
Latency 16865us 1188ms 725ms 26275us 35302us 135ms
Version 1.96 ------Sequential Create------ --------Random Create--------
test2 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
256 27477 56 349979 97 1115 1 28242 55 453860 99 857 1
Latency 779ms 631us 15595ms 785ms 79us 14245ms
1.96,1.96,test2,1,1356639819,8G,,477,98,39592,10,24128,4,2083,96,99561,9,606.1,17,256,,,,,27477,56,349979,97,1115,1,28242,55,453860,99,857,1,16865us,1188ms,725ms,26275us,35302us,135ms,779ms,631us,15595ms,785ms,79us,14245ms
Ubuntu 12.04 LTS na qcow2
test1# bonnie -d bonnie -u 0:0 -n 256
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
test1 8G 476 98 37107 5 29894 6 2132 96 878243 82 1365 40
Latency 16726us 1467ms 822ms 4232us 5511us 73827us
Version 1.96 ------Sequential Create------ --------Random Create--------
test1 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
256 26194 53 354900 99 1013 1 26470 52 453667 98 813 1
Latency 851ms 617us 15240ms 888ms 51us 16378ms
1.96,1.96,test1,1,1356655754,8G,,476,98,37107,5,29894,6,2132,96,878243,82,1365,40,256,,,,,26194,53,354900,99,1013,1,26470,52,453667,98,813,1,16726us,1467ms,822ms,4232us,5511us,73827us,851ms,617us,15240ms,888ms,51us,16378ms
Ubuntu 12.04 LTS na raw
test2# bonnie -d bonnie -u 0:0 -n 256
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
test2 8G 476 98 37935 10 24315 4 2202 98 896357 84 1310 39
Latency 16929us 827ms 856ms 3895us 1821us 70495us
Version 1.96 ------Sequential Create------ --------Random Create--------
test2 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
256 26764 55 359721 99 1052 1 27046 53 457961 98 803 1
Latency 806ms 641us 16137ms 858ms 66us 17820ms
1.96,1.96,test2,1,1356657989,8G,,476,98,37935,10,24315,4,2202,98,896357,84,1310,39,256,,,,,26764,55,359721,99,1052,1,27046,53,457961,98,803,1,16929us,827ms,856ms,3895us,1821us,70495us,806ms,641us,16137ms,858ms,66us,17820ms
