Recently I fixed my OpenVPN tunnel and figured out what was wrong with my bridge setup. Here is the complete setup that I have.

Server

I wanted to have as much of the configuration on the server as possible so that I could easily add more clients and wouldn't have a need to update client configuration when ever the server preferences change.

Here is my OpenVPN server configuration:

mode server
tls-server
dev tap0
server-bridge 10.5.6.1 255.255.255.0 10.5.6.200 10.5.6.209
ifconfig-pool-persist /var/run/openvpn-ip.txt
keepalive 10 60
ping-timer-rem
persist-tun
push "route 10.5.6.0 255.255.255.0"
push "dhcp-option DNS 10.5.6.1"
ca /root/easy-rsa-2.0/keys/ca.crt
key /root/easy-rsa-2.0/keys/ressukka.net.key
cert /root/easy-rsa-2.0/keys/ressukka.net.crt
dh /root/easy-rsa-2.0/keys/dh1024.pem
client-to-client
up /etc/openvpn/ovpn-ressukka.sh

and the up script:

#!/bin/sh
# Bind the tunnel interface to the bridge

BRIDGE=br0

ifup $1
brctl addif $BRIDGE $1

There is nothing really special about that configuration. The server is in TLS mode configures a bridge. The keys are generated with easy-rsa by following a openvpn howto entry. The up script just binds the tap0 device to the network bridge after bringing up the device.

Next I created the interface configuration by adding the following to /etc/network/interfaces:

iface br0 inet static  
 address 10.5.6.17  
 gateway 10.5.6.1  
 netmask 255.255.255.0  
 bridge-ports eth0

The trick here is to create a single bridge with just the eth0 device. We use the up script for openvpn to add the tunnel device to the bridge. Otherwise the bridge would never contain the proper devices.

Client

As for the client, you simply set the client to use the CA-certificate and Host key created with easy-rsa, set the hostname and tunnel type. Tunnel type is assumed to be a tun-device instead of tap, so in my case I needed to change it too.

There is no need to tell the client anything else. Everything else will be negotiated through the tunnel. And since I use NetworkManager to set up my tunnels I didn't have a need to drop in to a shell even once at the client.

After my post yesterday about locales and my problems, I got some comments.

Bryan, Simon and Wouter all pointed out what eventually figured out too but didn't mention in the post (I actually had to re-read the post to see what I wrote) that LC_ALL overrides all of the settings. Even though I figured it out eventually, with these comments it finally me that the purpose is actually to temporarily override the locale.

Another thing that didn't occur to me while figuring out the correct locale for my system was that I'm thinking about it all wrong. Since I've already gotten used to broken locales I kept thinking that I want en_US locale with some Finnish settings, but in reality I wanted fi_FI locale with English language. I have to admit that I was pretty sceptical about the solution but decided to try it out. And to my surprise, it actually worked!

So the final solution is this: LANG="fi_FI.UTF-8"
LANGUAGE="en"
LC_MESSAGES="en_US.UTF-8"
LC_TIME="en_US.UTF-8"

It is a pretty clean solution and I'm happy to live with it. And as Simon pointed out, administrator changes should be preserved through upgrades. If not, one should file a bug for it.

There we go, another problem solved. Thanks for the comments and suggestions.

Linux has this great thing called locales. You can basically control everything through a few system variables. The system is so flexible that you could in fact have English language with Swedish dates and Chinese error messages.

The problem with the current system is that it's controlled mostly with one variable. Just about everyone I know (including me) set the system locale to en_US because they don't want to use their local language as the system language. This in turn causes new problems, mostly due to the metric system used in the sane countries.

I finally got around to debug how to get the system to speak my language. The key here is the file /etc/default/locale which on Debian based systems contain the locale settings. Usually there is just one line in that file.

LANG=en_US.UTF-8

Since I live in Finland and want my computer to speak English I can add the following lines:

LC_NUMERIC="fi_FI.UTF-8"
LC_PAPER="fi_FI.UTF-8"
LC_NAME="fi_FI.UTF-8"
LC_ADDRESS="fi_FI.UTF-8"
LC_TELEPHONE="fi_FI.UTF-8"
LC_MEASUREMENT="fi_FI.UTF-8"

This replaces the settings for numerals (decimal separator and such), paper (yes, the rest of the world uses A4), name, address, telephone and measurement (YAY, metric system!). This way I have a nice English speaking system with Finnish settings.

I tried setting LC_ALL to fi_FI.UTF-8 but that causes gnome to speak Finnish to me, even though the LANGUAGE and LANG settings are set the English. LC_TIME is something I'd like to use, but I find the Finnish abbreviations for weekdays and months to be confusing. LC_MESSAGES causes gnome to talk partially Finnish, the general locale is English as it's supposed to be, but for example gnome-panel changes the menu entries to Finnish.

I wish that it was easier to set these settings. I also doubt that various tools know how to respect these settings and will overwrite that file with the default setting. That is why I'm writing this entry so that I remember how to fix it when that happens.

In the end the system is flexible but it's built in a funny way. The actual variable built by combining 2 or 3 values: language, country and possibly encoding. So to make thing easy for me I would have to declare en_FI.UTF-8 locale and start translating applications to that locale. I don't want to, so I'm sticking to this "temporary" solution.

I just have to say it, memory management is utterly broken in Linux. Luckily it's not as broken as it used to be. Without going in to too much detail, there is a feature called overcommit in Linux. What this means is that it will over allocate memory to applications. At first thought it sounds insane. Why would one want to over allocate memory. It's like eating a cake and having it too.

Well, there is a reason why this is done. If you think about resource allocation in general, there are a lot of places where this is being done successfully. For example ISPs always over sell their capacity because most of the users use a fraction of the resources allocated to them. That is why guaranteed capacity is so much more expensive, they literally allocate it for you (most of the times). So in a way memory management is like your friendly sales person from Linux corp. who is happy to sell what over you want because he needs the deal for his own bonus.

So it is in fact reasonable to overcommit memory? Bzzt! Wrong! There are always problems with over allocating. ISPs, for example, monitor the usage of their resources and when certain limit is reached they get more resources. With memory it's not so easy, you have a finite amount of memory in your computer. Swap comes to the rescue to a certain degree, but it is critical for the applications to know their limits. If you have 1 MB of RAM available for the application let the application know. In most of the cases the application will crash because it doesn't know how to deal with the out of memory situation, but this is a side effect of the overcommit. Programmers don't run in to this kind of problems and they don't have to deal with them.

So what happens when you hit the limit? In Linux there is this thing called oom-killer. oom-killer has just one purpose, to kill processes. While oom-killer has some clever heuristics built in to it, it will most likely end up killing the wrong program. So it's like a big game of core wars on your computer.

Generating locales...
  en_AU.UTF-8... done
  en_BW.UTF-8... done
  en_CA.UTF-8... done
  en_DK.UTF-8... /usr/sbin/locale-gen: line 238:  1971 Killed                  localedef $no_archive --magic=$GLIBC_MAGIC -i $input -c -f $charset $locale_alias $locale
failed
  en_GB.UTF-8... done
  en_HK.UTF-8... done
  ...

This is what happens on one of my hosts, there are not that many services running on this host and for some reason the right process gets killed. After turning overcommit off:

Generating locales...
  en_AU.UTF-8... up-to-date
  en_BW.UTF-8... up-to-date
  en_CA.UTF-8... up-to-date
  en_DK.UTF-8... done
  en_GB.UTF-8... done
  en_HK.UTF-8... done
  ...

Apparently locale generation is one of the things that is capable handling memory as it's supposed to be handled.

So how do you turn off overcommit? Sysctl is the way! Easiest way is to add the following to /etc/sysctl.conf and running "/etc/init.d/procps.sh force-reload":

vm.overcommit_ratio = 100
vm.overcommit_memory = 2

The second line is the magic line which turns off overcommit. Insanely enough the other options are 0 (Default) and 1 (Always overcommit), which just doesn't make any sense. If the default is to do overcommit why would I want to set it to always overcommit. The first line defines the maximum amount of memory a program can allocate. This is explained better on a post on fedora-devel.

Even with these changes, overcommit isn't completely disabled, but things are a lot better. While overcommit is built because it makes sense, it doesn't work. Lets hope things get even better in the future.

Update: Arthur de Jong pointed out that in certain situations overcommit is needed, like when there is a large application running on a server that prevents further memory allocations (thus disabling remote access and so on). That is true, I'm not trying to say that overcommit isn't useful in some cases. I'm just saying that it's just a disaster waiting to happen, unless you have finetuned the environment just right.

Way back when I created my LVM array for the NSLU2 I opted for USB Stick that would hold the Swap and /var/log so that the actual disks could spin down. At the time I gave the scheme quite a bit of thought because I knew that the stick would eventually fail. That is why I decided to put the data which mattered the least on the long run. Of course failing stick would bring down the whole host because Swap was there, but it was an acceptable risk. It's not like the host needs to be online 99.999% of the time =)

Today the stick started presenting with errors and since it's already been a year since I started using the stick, it's most likely breaking down. A new stick costs next to nothing and I happen to have plenty of them laying around. Time to make the switch. The good thing about picking LVM for the disks is that the process is really simple. One would think.

I plugged in the new stick, created partitions on it and ran pvcreate and vgextend to add the disk to the volume group. Next it was time to move the data.

pvmove -v /dev/oldstick /dev/newstick

After this it should be a matter of vgreduce and we should be all set. Well, it's not as easy as that. Usually this would be the case since not too many use LVM mirroring. The thing is that pvmove doesn't know how to move mirrored drives. Which normally isn't a problem. You just reduce the mirror and extend it to the new drive. But in my case the mirror log is the one that is blocking my move. LVM handles mirrors by writing a log partition which requires a 3rd drive for a normal 2 drive mirror. There is a way around this, by keeping the log in the memory which I didn't want to because of the limited RAM.

LVM is capable of converting between mirrored and unmirrored logical volumes quite easily, but it's time consuming process to re-create the mirror. It's not like I'm in a hurry, but it still seemed too much. The solution is to convert the log in to an in-memory log, removing the old drive and converting the logs back to persistent disk logs.

lvconvert -m1 --corelog /dev/vg00/Vol1
vgreduce vg00 /dev/oldstick
lvconvert -m1 /dev/vg00/Vol1

It's pretty much the same amount of work to run that instead of removing the mirror and recreating it.

As for me, I have a working array again. The old stick has been dealt with (it's no longer operational and life is back to normal.

A while back my NSLU2 setup broke down, the reason was that an external USB drive failed. The HD model had a known problem that caused the drive to fail. The drive was under warranty at the time, but I knew that the actual HD was most likely working just fine. After a bit of pondering I decided to void the warranty and took out the HD from the case. After a few scares, the drive worked correctly.

When I bought that drive, I was smart and bought a second drive for backup purposes.. But as usual, the drive wasn't connected to anything and didn't have any backup value. So I had to set up the mirroring if I was to avoid future crashes, maybe I wouldn't be so lucky next time.

When I set up the software raid, I went with plain LVM mirroring. Almost all documents in the web talked about mdadm + LVM, but that would make my setup more complex and I didn't want to go that way with the limited CPU and RAM of the NSLU2. Setting up plain LVM mirroring was almost too easy, the only caveat was that for plain 2 drive mirroring you need 3 drives in the Volume Group. This wasn't a problem for me since I already had an USB Stick acting as the /var of the box (this allows the HDs to spin down).

So the complete process for adding a mirroring with LVM was just:

pvcreate /dev/sdc1
vgextend vg00 /dev/sdc1
lvconvert -m 1 vg00/Vol1

LVM will choose the drives it needs to use for the mirroring and start copying data. You can monitor the progress by doing

lvs -a -o +devices

That's all there is. Now, this post was supposed to be about recovering data from a broken volume group. A few days ago, the same drive took a dive from the table (maybe it figured out that it was on extended time already) and broke down (yes, we all know that clicking sound).

The NSLU2 hung and I quickly figured out that the only thing that wasn't mirrored was the data on /var and Swap. /var was on a USB Stick, but the Swap happened to be on that broken drive. OK, so kernel doesn't handle disappearing swap too well (at least if you have 32MB of ram). So I tried to reboot it, with little luck.

LVM isn't too helpful when it comes to broken drives. It will work just fine if you don't deactivate the Volume Group, but it refuses to activate it before it's fixed. I moved the working drives to my laptop and ran lvs to see the situation (udev already set up the VG for me)

lvs -P -a -o +devices

The -P flag means partial, so lvs includes the status of those Volume Groups which are missing physical volumes. The list verified that the Swap was actually on the broken drive but everything else was either mirrored on a working drive or on the stick. Removing the missing drives fixes the Volume Group and allows the drives to be mounted read/write.

vgreduce --test --removemissing vg00
vgreduce --removemissing vg00

The first run is just to see that it's actually capable of doing something sane. It will complain about missing drives on both run though.

Finally after the Volume Group was fixed I just ran fsck on all the volumes just to make it easier for the NSLU2 to boot. One of the partitions did have corruption, but I think it has been corrupted since the drive initially failed, it just wasn't scanned until now.

To remove the drives cleanly without reboot you need to run the following:

vgchange -a n vg00
sync

Then you can just unplug the drives.

Initially i made the mistake of exporting the VG before disconnecting. I wanted to make as clean exit as i could, but exporting the VG will change the state on disk and the NSLU2 will not boot before the VG is imported again.

Every now and then we all see these "Year of the Linux" posts and ignore them. Most of us have been around for long enough not to trust or make those claims. Today I stumbled upon a post by Melissa Draper, which curiously enough started with similar words. She mentions that we are nearing the tipping point and refers to the recent OEM interest in Linux.

I for one still say that we are not there yet. During last week I've been struggling with printing. A Lot! I acknowledge that printing in Linux has taken major leaps from the times of basic lpd and the like. Still it appears that printing needs a lot more work.

For example, GIMP has its own printing dialog which forces you to manually configure each printer. While it's an easy job for someone familiar with the printing system it's not something a normal user is able to do. The printing process is way too complicated, most users just want to resize the image a bit to print. Please offer a simple printing dialog for them.

The problem isn't just with GIMP. Another example is Firefox which has a clean little printing dialog. The problem is that it assumes paper size and even with the proper paper size I can't manage to print a page without parts of the page going outside of the margins.

OpenOffice.org appears to default to letter as the default paper size. This means that both the default document is edited as letter and when printing the printing system assumes you are printing to letter even if the page you are printing is A4. Other than that, it manages to print without glitches.

I know that these are just a few applications, but think about it. These applications are amongst the most used applications on Linux. These applications represent how Linux works for normal users. Users are quite often naive, they make bad associations. Bad experiences with a few of the key applications make a bad impression on the whole effort. Also, bad news travel faster than good news...

In case this sounds like I'm just bashing here, I have to mention that things like CUPS are a huge effort to solve these problems with the right kind of solutions. Also, while testing around I successfully printed out of evolution, gnome-dictionary and various other applications that use gnome printing libraries and they printed just fine without any problems. I'm sure that there are similar efforts on KDE land (and elsewhere) so we are moving to the right direction. The problem is that there is a long way to go before we can claim world domination.

Quite some time ago I used to host update.ressukka.net which was a service for certain Jabber clients that allowed the clients to query the latest released version and offer it to the client without using any other protocol than XMPP. It was a useful service at the time, but over time it became easier to just do the same thing over HTTP (since the download was done through HTTP too)

Over time I've noticed mail coming in to that domain, which is kind of natural since mail harvesters quickly pick up Jabber IDs as e-mail addresses since they use the same format. Recently I received a massive scan for to that domain and since it's not hosted on the server that receives my mail, it gets queued for delivery. This is where things go wrong.

Mails for that domain should never be queued in the first place. I've configured my mail server to accept mails to ressukka.net. Apparently historically postfix has accepted mails for subdomains automatically, so if you configure mydestination to "localhost" mails are accepted for "invalid.localhost" too. Apparently the devs have noticed this problem and have added an option to control this behaviour. There is an option called parent_domain_matches_subdomains (how logical is that?) that allows you to define which features behave this way.

The fix for me was to change the default setting:

parent_domain_matches_subdomains = debug_peer_list,fast_flush_domains,mynetworks,permit_mx_backup_networks,qmqpd_authorized_clients,relay_domains,smtpd_access_maps

in to:

parent_domain_matches_subdomains = debug_peer_list,fast_flush_domains,mynetworks,permit_mx_backup_networks,qmqpd_authorized_clients,smtpd_access_maps

By removing the relay_domains setting I was able to limit incoming mails to just the domains that are actually on the server.

Problem solved? Not quite...

I still have ~2500 mails waiting to be delivered to update.ressukka.net (and various other removed hosts). Luckily there are tools out there to remove certain mails from the postfix queue. By (ab)using the script mentioned in Postfix: The definitive guide (download script) I was able to easily clear the offending mails from the queue. The trick was to use a regexp as the mail address for the script.

./pfdel.pl .\*@update.ressukka.net

In any other script I would call this a security problem, but here it's a feature =)

Today, while browsing through a list of the packages updated in Gutsy, I ran in to backupninja. It's interesting to see how many packages I discover just by reading the list of updated packages.

Backupninja so far looks pretty good. I like that it's made in such a way that it's extendible and the fact that it takes advantage of other tools that already perform well, but lack in features. The only thing it lacks is the ability to handle offline situations.

Even though backupninja lacks the ability to handle offline situations, I think I'll go for it. For this same reason, I'm pretty much abandoning the [[!comparison table|2007/backup_comparison]] I started to create. I'll update the same information about backupninja to it, but unless I don't like backupninja for some reason it's unlikely that I'll revisit it any time soon.

From time to time I try out new distributions that I see pushing out new releases. And almost as often I come to wonder why did they release in such a state.

There are quite a few distributions which can't really decide if they are trying to be a server or desktop distribution. Usually this causes the Desktop version to be riddled with problems that are normal on servers but should never appear on a desktop.

Lets pick an example: I installed CentOS earlier this week which intended to install it as a desktop installation. What annoyed me was the need to use a separate root password to manage the computer. This is an issue that divides opinions pretty evenly. Others like to enter their root password and others like the sudo approach. If you think of the normal users, they have a hard time remembering more than one password so making them enter the root password is insane. Installation asks quite a few questions, questions that a normal user shouldn't need to know. If you want to know in more detail, install CentOS and see for yourself.

The problem here, as I see it, is that most distributions try to do 2 things at the same time. They try to provide an enterprise server distribution and a desktop distribution. There are some distributions that just simply aim to please the desktop users but hardly any that only aim for the server platform. Usually the ones that work towards a server environment push in the desktop side too.

So far Ubuntu is the only distribution that I've tried that is able to pull off both. It's a killer desktop distribution as well as a server distribution. I think it's all about setting goals and keeping them. Rational goals make a huge difference.

Distributions should stick with what they do best. It's not about trying to make it happen. It's about admitting that you can't do everything and working within the resources you have.