xgrr.de – the whole not the half

things, thoughts and stuff out of life, daily business and computer science


Virtualized Networks – Next Generation Networks [Update]

In professional areas virtualization of servers and workstations enables us to evaluate operating systems and software packages in an isolated environment and without great costs. But this is still the starting point of virtualization. As our current internet topology and software applications ask for integration – the Cloud is a good keyword here – we have to be not only able to virtualize physical PC hardware but also provide network environments which are abstracted from our current internet.

This would enable network engineers, system administrators and of course scientists to easily develop new network infrastructures, test distributed deployments or invent new protocols on top of the network stack. The impact of a commercial virtualization technology for networks would have a huge impact of the way we currently build networks.

With the development of NGN (Next Generation Networks) this idea is pushed by a few selected but until now it did not surface to the broad public.

Update: I made a bit of research on the topic. Nothing deep really but I still found some interesting material:

I also found a lot of sites talking about network virtualization in connection with classic computer virtualization. This is only a part of what is meant. Network virtualization should be an end to end implementation with virtual nodes on the end points itself in the form of a software or firmware on the NIC and also on the nodes in between. It has not necessarily be linked to virtualized servers although this would of course make a lot of sense.

Hyves: 1700 servers running with no ideas

Recently I visited the social network Hyves which is very popular in the Netherlands and saw and job offer for a system engineer/administrators position. In the comments to this position there was harsh critics about the network of Hyves which consists currently of about 1700 servers running Gentoo Linux.

When I read more about the infrastructure I realy wondered why they need 1700 which results in A LOT of money for operation and maintanance. I’m not the only one who said this but the tone was different than my :) . I saw a thread about virtualization which is working very well recently for non real-time applications.  And Apache, MySQL or whatever they use as a backend is not real-time.

However they reason the choice for a lot of servers with the need for I/O throughput (when you read my blog closely you will see that I/O performance is the big problem when working with virtualization). What I wondered so much is that no post mentioned the use of SAN. They certainly have something similiar otherwise I don’t want to know how they store all the data they gather. So why don’t they use it for virtualization? Good SAN provide a lot of needed I/Os as well as redundancy. When using in combination with NFS (very upcoming – SANs are mostly connected by using iSCSI) the bottleneck of low I/O operations is decreased.

I for myself run VMware very successfully. With a good plan and VMware Infrastructure 3 or something similiar a consolidation of the needed servers can be done and the overall costs decreased.

VMware I/O tunings

When I realized that the VMware hits the I/O limits I searched for some tweaks to better the situation. In the VMware Knowledge Base are some nice tweak which can make the situation less worse. Here the best one:

VMware is using the shaed memory on the Linux host. Normally the SHM is limited by the default settings of the Kernel or your settings in the configuration. This can be very limited. According to the Knowledge Base SHM should have at least the size of your memory. So if you have 4 gigs of RAM make it 4 and give some extra so 5 gigs. You can change the value on the fly by remounting /dev/shm:

mount -o remount,size=5G /dev/shm

To make this persistent you can change the /etc/fstab. Under Debian you have to change /etc/default/shm and give the right bytesize there.

VMware hitting limits

I’m busy with VMware in production since a few months now. Every day I get to know the system a little better. Just today I identified the biggest bottleneck when running a few virtual machines on a “normal” server. My setup uses about 4 gigs of RAM with on a 300GB Samsung SATA II disk. The system is driven by an Opteron 1212.

With 8 running virtual machines one of the cores is completely wasted with waiting on the I/O operation. The normal system load is very low but when you look carefully you will realize that the system is under high I/O load even if the virtual machines are not under high load.

Cpu0  :  0.0%us,  2.5%sy,  0.0%ni,  0.0%id, 96.2%wa,  0.0%hi,  1.3%si,  0.0%st
Cpu1  :  0.0%us, 11.5%sy,  0.0%ni, 88.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

In the snip above taken from top in the host you can see that Cpu0 is completely busy with waiting on I/O. The harddrive is the bottleneck for VMware server which will host more than 5 machines.

I will try to get a second harddrive into my machine an create a sofware RAID. Maybe I can produce more I/Os per second and can take the load from the CPU. This would better the overall performance of the system.

Convert Virtualbox .vdi to VMware .vmdk

I love VirtualBox because is fast and without a lot overhead. But network bridging is kind of pain in the ass with VirtualBox . I hope they change this fast. Anyways I had to convert my VirtualBox images to the VMware file format. This is easily done with qemu-img a nice tool bundled with qemu. It can handle serveral file formats:

 Supported format: qcow2 vvfat vpc bochs dmg cloop vmdk qcow host_device raw

So you could convert some other stuff too. Converting is easily done (this example is for Windows by using qemu-0.9.0-windows):

 qemu-img.exe convert -O vmdk hdd.vdi hdd.vmdk

In this case we convert the disk hdd.vdi to hdd.vmdk. The input format is detected automatically but can be overriden by adding -f switch. Output is vmdk as given with the -O switch.


Some VMware related I/O speed optimizations

While studying the documentation to Lustre (a distributed Linux filesystem) I found this:

Choosing an alternate scheduler is an absolutely necessary step
to optimally configure Lustre for the best performance. The “cfq” and “as” schedulers should never be used
for server platform.

So I revised my VMware installation and found out that all my guest are configured to use the deadline scheduler. But as it seemed my host did use the cfq. In the user guide there was described how to change the used scheduler while running the system (where hda is the disk):

[root@cfs2]# cat /sys/block/hda/queue/scheduler
noop [anticipatory] deadline cfq
[root@cfs2 ~]# echo deadline > /sys/block/hda/queue/scheduler
[root@cfs2 ~]# cat /sys/block/hda/queue/scheduler
noop anticipatory [deadline] cfq

So I changed the scheduler and it seems the overall performance is dramatically increased.