xgrr.de – the whole not the half

After I wrote my post about the remote WebDAV access to files, I tried around with some programs which integrate the WebDAV mount as a network share. Unfortunately this isn’t working to good for large files. All the programs I tried first downloaded the file and then started to access it (e.g. playing a mp3 wasn’t that nice if you have to wait a minute between each song)

So a new solution needs to be found. A long time ago I had netjuke up and running. But it seems that this projekt is dead for quite a while. A little search brought me onto mp3act. Which is similiar to netkuje but AJAX based. It also brings integration for Amazon Web Services to download album art.

Yeah. Streaming isn’t that compatible with firewalls around the world especially in corporate situations. So I needed to figure out a better solution to get to my precious files without hassle. The solution: WebDAV.

I allready use WebDAV in combination with Subversion. DAV is an extension to the HTTP protocol which allows you also to upload files onto the webserver. This makes FTP useless and with some tricky tools it also integrates quite nicely in your desktop environment.

I have a 2 tier setup. At home my server features a Apache2 with enabled WebDAV. Configured as follows:

Alias /share /path/to/your/files

<Location /share>
	Header add MS-Author-Via "DAV"

	BrowserMatch "^WebDAVFS/1.[012]" redirect-carefully
	BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
	BrowserMatch "Microsoft-WebDAV-MiniRedir/5.1.2600" redirect-carefully
	BrowserMatch "^WebDrive" redirect-carefully
	BrowserMatch "^WebDAVFS" redirect-carefully

	DAV on
	AuthType Basic
	AuthName Share
	AuthUserFile /path/to/your/.htpasswd
	Require valid-user
</Location>

And on my Windows XP PC there is a nice tool from Novell running called NetDrive. As this tool could not be provided by Novell directly due to license issues I put it here for download (they especially mention to look for netdrive.exe on Google. So I think it shouldn’t hurt ).

With NetDrive you can mount the share like a network share as a normal drive and also can access it as such. With this solution you have all-time access to you files without need for any kind of strange solution which can issue other problems like NAT for instance.

For my server at home I want a clean media streaming solution not involving any kind of webserver. VLC is capable of doing all kind of streams. For video on demand only RTSP is usefull. In combination with the HTTP interface for VLC this becomes quite a nice streaming server for all kinds of media stuff.

One big problem this solution has of course. RTSP or better RTP is not made for NAT. So the actual media is not transmitted through a firewall. I did some extensive testing with my VLC at home. There is actually no possibility to force VLC to use a specified client port. With all my tests VLC ended up using a port starting from 2500 udp. I didn’t play it through and tested where it’ll end.

This means your clients or viewers need to configure their firewall according to this port range. VLC does not support any kind of proxy or STUN so this is the big problem.

Here is how I run my VLC (Server). Note that the -I http switch starts the VLC webinterface on port 8080.

vlc -vvv --color -I http --rtsp-host 0.0.0.0:8554

Here is how you point the viewers VLC client to the stream.

Some time ago I placed a post (see Starten von Linux ohne Passwort) on how to boot without knowing any password of the machine. The post describes how to do it with Lilo. In the mean time I found out that some derivates behave quite different and that Grub more and more becomes standard boot manager (poor Lilo).

I tried it with some derivates and it worked like a charm. Please note that there could be differences between the server and possible available community versions.

For all derivates that operates with Grub counts: As soon you see Grub stop boot process and change the boot parameters according to the following small table:

Derivate			| Option
=============================================
SuSE Linux Enterprise Server 10	| init=/bin/sh
Red Hat Enterprise Linux AS 4.5	| single
Debian GNU/Linux 3		| init=/bin/sh
Debian GNU/Linux 4		| init=/bin/sh

If you hold on to that you should be able to boot directly into a shell. Remount the / partition and change any password you like. There are some tools which try to prevent that from happening but as far as I’ve experienced it nobody deploys them (except for Red Hat but you’re still able to access the shell via the “single” switch)

Let met know if you make other experiences or discover other switches for different derivates.

If I had known this before I wouldn’t have done it. Yesterday I decided it is a good thing to switch my server to a RAID so I started late at night backing up all the data (~320GB) to another hard drive. Then I installed the RAID and started to mirroring all the data to the newly created RAID. Unfortunately the system didn’t boot and my hosting partner took his time to select another kernel to boot. So in the end I stayed up about 36 hours with about two hours sleep only to wait about in total 9 hours that the guys select another kernel. I’m totally stressed out now…

Without comment:

With my old company I still maintain the email services. The mail load increased over the last year so the server itself isn’t able to handle both email handling and spam tagging. For that purpose I started to deploy a dedicated Spamassassin server which didn’t perform well in the beginning. A second server with the same configuration was installed to help the first one.

Later on it turns out that the networking stack of spamc is not scaling very well as we suffered a lot of connection resets and a bunch of mails were not scanned at all due to connection losses. I started to think of a better load balancing and started to play around with LVS which turns out to be very complicated.

I remembered a fine program which can handle HTTP and TCP load balancing and is acting as a transparent proxy: HAProxy. With this load balancer installed on the mailserver both Spamassassin machines are balanced depending on their actual connections which ensures a better connection handling than simply DNS round robin.

I couldn’t believe it. I wanted to install daemontools a package programmed by Bernstein primarily to use with qmail. In the qmail “handbook” “Life with qmail” are a lot of tricks to patch this external tools. So even when I don’t install qmail but these external tools I check this site. When I downloaded the netqmail package I was astonished. After a few years of version 1.05 there was 1.06 released. Even better I found a site from Bernstein declaring that all the packages are really Open Source now and NOT prohibited from redistributing with modified source. This is a big happening for the qmail community because all the nice things can be included in one source now without the need of incompatible patches.

I wondered if the count of 1700 servers is really realistic. I checked Wikipeda. They currently run 349 servers to be on position 9 on the Alexa list. Just wanted to say that 1700 servers for a small site (compared to Wikipedia Hyves IS small) is a little too much. Better not use Gentoo in production environment

Recently I visited the social network Hyves which is very popular in the Netherlands and saw and job offer for a system engineer/administrators position. In the comments to this position there was harsh critics about the network of Hyves which consists currently of about 1700 servers running Gentoo Linux.

When I read more about the infrastructure I realy wondered why they need 1700 which results in A LOT of money for operation and maintanance. I’m not the only one who said this but the tone was different than my . I saw a thread about virtualization which is working very well recently for non real-time applications.  And Apache, MySQL or whatever they use as a backend is not real-time.

However they reason the choice for a lot of servers with the need for I/O throughput (when you read my blog closely you will see that I/O performance is the big problem when working with virtualization). What I wondered so much is that no post mentioned the use of SAN. They certainly have something similiar otherwise I don’t want to know how they store all the data they gather. So why don’t they use it for virtualization? Good SAN provide a lot of needed I/Os as well as redundancy. When using in combination with NFS (very upcoming – SANs are mostly connected by using iSCSI) the bottleneck of low I/O operations is decreased.

I for myself run VMware very successfully. With a good plan and VMware Infrastructure 3 or something similiar a consolidation of the needed servers can be done and the overall costs decreased.