xgrr.de – the whole not the half

For years now I was using PHP as a Apache module and not thinking twice about it. Easy to install not much to maintain and working reasonably well. Server too slow? Buy a new server. Recently one of our servers got really slow again with varying timeouts while serving pages. Memory shortage. No new server available I needed to make the most of it and started investigating how to decrease the memory footprint of Apache.

FastCGI was the answer. With PHP as a module installed every Apache process can take up to the maximum amount of memory you defined in php.ini plus what the Apache process needs by himself even though this process is only serving static content. With the switch to FastCGI it was possible to use the threaded version of Apache (mpm_worker). Static contents are now served really fast without need for invoking PHP at all. PHP scripts are served by calling the FastCGI IPC.

As memory is still low Apache and PHP are both configured to “die” after quite a low number of requests served. This can sometimes lead to a small delay while the processes are restarted but ensures that the memory footprint is kept low.

How did I do it? I won’t go into details on how to install Apache, mod_fcgid and PHP because there are loads of howto’s out there. Please find the configuration I’m using below. I’m using Debian and Apache, mod_fcgid and PHP are installed out of the repository. If you have a custom compiled version or different distribution your paths can differ.

Depending on your available hardware it is feasible to tweak the settings for the mpm_worker module so more concurrent clients can be served and/or the amount of requests which is handled before re-creating the thread is higher. It is also very recommended to use eAccelerator in conjunction with this setup. See my post on the topic for more information.

/etc/apache2/apache2.conf (excerpt):

<IfModule mpm_worker_module>
	StartServers           4
	ServerLimit            4
	MaxClients           128
	MinSpareThreads        8
	MaxSpareThreads       16
	ThreadsPerChild       32
	MaxRequestsPerChild  500
</IfModule>

/etc/apache2/mods-available/fcgid.conf:

<IfModule mod_fcgid.c>
	AddHandler fcgid-script .fcgi .php
	FCGIWrapper /var/www/php-fcgi-starter .php

	IdleTimeout		3600
	BusyTimeout		 300
	ProcessLifeTime		7200
	IPCConnectTimeout	  10
	IPCCommTimeout		 360
	MaxProcessCount		  15
	MaxRequestsPerProcess	  -1
	PHP_Fix_Pathinfo_Enable    1
</IfModule>

/var/www/php-fcgi-starter:

#!/bin/sh
PHPRC=/etc/php5/cgi/
export PHPRC
export PHP_FCGI_MAX_REQUESTS=250
export PHP_FCGI_CHILDREN=1
exec /usr/lib/cgi-bin/php

Okay you might think there was a raid at some strange gang called PSU. You could not be more wrong . I actually fixed my strange RAID problems this weekend by replacing the PSU with a really good one. I chose the Thermal Take Toughpower XT 750. It comes with cable management, a fancy 14cm silent fan and enough power.

The last few days I was desperate to fix my server. A drive was crashed and I ordered new ones to replace the “broken” one but it didn’t work. All new drives were failing too. In my desperation I ended up bugging the RAID vendor but they needed more information during the weekend and that was simply not fast enough for me.

Last friday I decided to replace the crappy old PSU with a fancy new one. And it did good. No more drives failing, due to the nice fan the temperature of the disks kept on falling compared to before. Overall a good result. Finally I can access my precious data again

It’s always the same. Your machine/workstation/server is running endlessly without problems. Then when you move and stupidly decide to store your backup drive with all the other stuff at a friends place a hard disk fails and you’re without hope.

Of course exactly this scenario happened to me just last week. So I ordered two new drives for my RAID and if nothing goes wrong can upgrade the overall capacity to 4.6 or even 5.5 TB. And that at home with consumer hardware. I love this world.

Still first I’ve to fix the RAID which is curiously crashing every time one drive fails. But I won’t give up the hope.

You know how it is. Just created a new partition but it is not directly regocnized by the system without a reboot. And rebooting means downtime, going to the data center thus inconvenience. There are a few tricks you can try before having to move your body physicly:

partprobe

Check if the new partition was recognize using dmesg. If that didn’t help try the following:

hdparm -z /dev/sdX

Where /dev/sdX is the device on which you created the new partition.

For my thesis I needed to get some stuff converted to not look silly for the PDF output of pdflatex. The tools I found were mostly really crappy or not capable of doing what I needed to be done. A simple tool helped me to convert a lot of SVG to PDF for including in my LaTeX document.

I followed this small tutorial to compile the application. On my Debian system I needed to install two libraries to get the compiling started:

apt-get install librsvg2-dev libcairo2-dev

I attached my compiled version of svg2pdf for x86_64 architecture.

Short version: Pain in the ass in the beginning. Long version. Areca provides a HTTP proxy and a CLI interface to the the controllers. Unfortunately the Kernel driver which comes with the vanilla Kernel from kernel.org does not include the necessary functions to allow the newest versions of these nice tools to access the controller directly, which is kind of stupid.

However. As my RAID was migrating and I wanted to have the progress (takes some time with 5TB) accessible nicely remote on my browser I searched the Areca FAQ section (which is really good btw) and found an entry which describes the problem and also provides a solution…

Only the version 1.71 of the HTTP interface is capable of connecting to the driver provided by the vanilla Kernel. I actually don’t know if they changed something in the driver in the meantime.

You can find the right version of the HTTP interface on the Durch Areca mirror.

During my struggles with the partition table of my RAID, I accidentily deleted partitions on the wrong disk. Very stupid as one of the partitions was part of a LVM Volume Group storing my original data. I tried several tools to recover my partitions.

parted, gpart and rescuept all were not able to get the right partition table back. Then I read something about testdisk. Testdisk was not only able to recover my primary ext3 and swap partition but also recognized the LVM2 partition. And the most usefull feature of testdisk is a progress indicator which is missed painfully with the other tools.

You might have noticed the anouncement of Googles new project today: Native Client. In short NaCl is a bridge between the web browser and the computing power of the clients pc. With this bridge it is possible to access the resources of the pc and use them for creating much more CPU intensive web applications than before.

The bridge is implemented as a browser plugin compatible with the NPAPI which features the most current browser except for Internet Explorer. Native Client implementes a redundant sandboxing mechanism to ensure that the module, seen as untrusted code, can do no harm to the underlying operating system. The sandboxing is implemented via various methods described detailed in the research paper published by Google.

With Native Client it is possible to implement applications which can take advantage of the full computing power of the underlying hardware. As sample applications Google implemented Quake I and a fractal application which is able to zoom in and out of the fractals. This shows that this framework offers extensive power to the application.

Heise.de the newsticker of the Heise Zeitschriften Verlag which also publishes c’t, iX and Technology Review described this new piece of technology as: “Browser as the next operating System“. Partly this is certainly true, as the normal boundaries of legacy web applications are blurred. The target market is certainly there. Virtually everybody is using a browser at least once a week. Web based applications such as Photoshop Express can use the power of the clients pc to calculate image operations and so on.

This move was expectable from Google after launching Chrome a few months back. And a lot of the members of the NaCl project are also assigned to the Chromium project on Google Code.

I’m very curious how this technology will evolve and how the acceptance is as there are certain security concerns when browsers or code from untrusted sources can access hardware directly. Even the recent heated discussion about bugs in CPUs can influence the security not only of the standard operating system but also of the untrusted code running in the Native Client.

Last week MySQL 5.1 was finally anounced. Now the MySQL founder Michael “Monty” Widenius commented on the release. He said that this version is crap. It contains a lot of bugs which can trigger data inconsistencies, system crashes or might be used as an attack vector on the MySQL infrastructure. In MySQL 5.1 a five year old bug is included which made it even to the Wikipedia site for MySQL. His explanation for this disastrous situation was too early change to release candidate status instead of pushing short releases as alpha or beta.

His solution for these problems: Change the development and release model. Let the community drive the development so that quality software is created. He mentioned the models used by PostgreSQL and Drizzle.

Drizzle is a code fork of MySQL and not owned by any company. Knowing the open source world a bit, this could be the answer to a lot of uncertainties which came up when Sun bought MySQL. But this depends on how the Drizzle developers doing their work.

All my mailservers are running Courier very successfully. It’s reliable most of the time until now I had almost no complains about anything. TLS/SSL support works fine, even with self signed certificates. The reason why I use Courier is that it has Vpopmail integration which I use in junction with qmail. So thats a mandantory requirement.

Quite a whileago I heard of Dovecot. And I was very pleased to also hear that it brings out of the box support for Vpopmail as user/password backend. Then I forgot about it for quite a while until a week ago when I finally decided that it is time to check it out.

Before I start: It was the best thing that I did to my self for quite a while. The compilation is quite easy and fast (compared to Courier where ./configure is recursing through all modules even you disabled them). I dropped it in a very simple deamontools script and I was good to go.

TLS/SSL support out of the box. Vpopmail support without hassle (I had absolutely no problems with authentication). Only the namespace was something not working. So if you don’t want you customers to reconfigure their clients simply change the namespace configuration of Dovecot (which is very easy) and your replacement is ready.

One small drawback I didn’t look at due to a lack of time is quota. The most functions extensions in Dovecot are implemented using a plugin interface so is quota. I didn’t check out how exactly this is done but judging from the overall experience I would say it should be straight forward.

But let’s talk some numbers. Of course I didn’t run a stress test or benchmark on my production servers but I am subscribed to the Debian mailinglist for quite a while now and have about 24k emails in one directory. So I compared the re-indexing with both Courier and Dovecot and Dovecot was faster and more responsive than Courier during the whole test even when accessing other mailboxes from the same connection.
The webmail system my other key component in my (subjective) test was also a bit faster on initial connection to the mail server and much faster afterwards when browsing through the folders as Dovecot implements some kind of indexing method.

During the next weeks I will migrate to Dovecot completely and remove Courier from my systems as I’m very pleased with this test result. Scripts for configuration and for daemontools are available on request.