Recently I fixed my OpenVPN tunnel and figured out what was wrong with my bridge setup. Here is the complete setup that I have.

Server

I wanted to have as much of the configuration on the server as possible so that I could easily add more clients and wouldn't have a need to update client configuration when ever the server preferences change.

Here is my OpenVPN server configuration:

mode server
tls-server
dev tap0
server-bridge 10.5.6.1 255.255.255.0 10.5.6.200 10.5.6.209
ifconfig-pool-persist /var/run/openvpn-ip.txt
keepalive 10 60
ping-timer-rem
persist-tun
push "route 10.5.6.0 255.255.255.0"
push "dhcp-option DNS 10.5.6.1"
ca /root/easy-rsa-2.0/keys/ca.crt
key /root/easy-rsa-2.0/keys/ressukka.net.key
cert /root/easy-rsa-2.0/keys/ressukka.net.crt
dh /root/easy-rsa-2.0/keys/dh1024.pem
client-to-client
up /etc/openvpn/ovpn-ressukka.sh

and the up script:

#!/bin/sh
# Bind the tunnel interface to the bridge

BRIDGE=br0

ifup $1
brctl addif $BRIDGE $1

There is nothing really special about that configuration. The server is in TLS mode configures a bridge. The keys are generated with easy-rsa by following a openvpn howto entry. The up script just binds the tap0 device to the network bridge after bringing up the device.

Next I created the interface configuration by adding the following to /etc/network/interfaces:

iface br0 inet static  
 address 10.5.6.17  
 gateway 10.5.6.1  
 netmask 255.255.255.0  
 bridge-ports eth0

The trick here is to create a single bridge with just the eth0 device. We use the up script for openvpn to add the tunnel device to the bridge. Otherwise the bridge would never contain the proper devices.

Client

As for the client, you simply set the client to use the CA-certificate and Host key created with easy-rsa, set the hostname and tunnel type. Tunnel type is assumed to be a tun-device instead of tap, so in my case I needed to change it too.

There is no need to tell the client anything else. Everything else will be negotiated through the tunnel. And since I use NetworkManager to set up my tunnels I didn't have a need to drop in to a shell even once at the client.

I've been meaning to write up on this for quite some time now, but always have something seemingly more important things to do.

Most modern Linux distributions offer ways to manage network interfaces via some kind of abstraction. Usually this abstraction allows one to dynamically rename and add network interfaces. For Debian family management is done with ifupdown, while with RedHat family it's sysconfig. In addition there is NetworkManager which is a cross platform solution for dynamic network configuration (which eliminates the need to rename interfaces).

Case Xenserver

Usually this comes in rather handy, but on other occasions it can be a pain. A while I was helping out with an installation of a XenServer instance. This server had, for some reason, ethernet interfaces reversed in comparison to the other XenServers on the site. Luckily the service console has the network configuration in ifcfg scripts. We were easily able to reverse the interfaces by binding the interfaces to certain hardware addresses.

The only problem is that the interface is renamed only when the interface is brought up. What's worse the interfaces were enumerated before the server would bring up all of the interfaces. Only eth0 was brought up for management purposes before enumerating. This means that the original eth0 was renamed to _tmp_xxxxxx. The (oh so elegant) solution was to create a script that does ifup eth1; ifdown eth1. Problem solved.

Case tap0

I also experienced similar problems when I was setting up my OpenVPN tunnel. I wanted to use a bridged connection to my network, but for that I needed to create a br0 interface with tap0 as a member.

It's easy enough to create ifupdown configuration to set up tap0. Hook that up to br0 and you are all set. The same problem will bite you here. The interface tap0 is actually created only when the tap0 is brought up and br0 members are added only if they are present.

The solution? In my case, create a custom script that adds the device manually after it's been created properly. I was unable to find anything that works without using a custom script =(

In retrospect

Looking back at these cases, I should have known better. The problem was obvious and it took me way too long to figure out the cause for my problems. Then again, the utilities should be able to create a reasonable abstraction for themselves that they use to determine the actual status of the whole system. This way the trickery needed for setting up simple interface would be obsolete.

You can't have it all, but you can always hope.

After my post yesterday about locales and my problems, I got some comments.

Bryan, Simon and Wouter all pointed out what eventually figured out too but didn't mention in the post (I actually had to re-read the post to see what I wrote) that LC_ALL overrides all of the settings. Even though I figured it out eventually, with these comments it finally me that the purpose is actually to temporarily override the locale.

Another thing that didn't occur to me while figuring out the correct locale for my system was that I'm thinking about it all wrong. Since I've already gotten used to broken locales I kept thinking that I want en_US locale with some Finnish settings, but in reality I wanted fi_FI locale with English language. I have to admit that I was pretty sceptical about the solution but decided to try it out. And to my surprise, it actually worked!

So the final solution is this: LANG="fi_FI.UTF-8"
LANGUAGE="en"
LC_MESSAGES="en_US.UTF-8"
LC_TIME="en_US.UTF-8"

It is a pretty clean solution and I'm happy to live with it. And as Simon pointed out, administrator changes should be preserved through upgrades. If not, one should file a bug for it.

There we go, another problem solved. Thanks for the comments and suggestions.

Linux has this great thing called locales. You can basically control everything through a few system variables. The system is so flexible that you could in fact have English language with Swedish dates and Chinese error messages.

The problem with the current system is that it's controlled mostly with one variable. Just about everyone I know (including me) set the system locale to en_US because they don't want to use their local language as the system language. This in turn causes new problems, mostly due to the metric system used in the sane countries.

I finally got around to debug how to get the system to speak my language. The key here is the file /etc/default/locale which on Debian based systems contain the locale settings. Usually there is just one line in that file.

LANG=en_US.UTF-8

Since I live in Finland and want my computer to speak English I can add the following lines:

LC_NUMERIC="fi_FI.UTF-8"
LC_PAPER="fi_FI.UTF-8"
LC_NAME="fi_FI.UTF-8"
LC_ADDRESS="fi_FI.UTF-8"
LC_TELEPHONE="fi_FI.UTF-8"
LC_MEASUREMENT="fi_FI.UTF-8"

This replaces the settings for numerals (decimal separator and such), paper (yes, the rest of the world uses A4), name, address, telephone and measurement (YAY, metric system!). This way I have a nice English speaking system with Finnish settings.

I tried setting LC_ALL to fi_FI.UTF-8 but that causes gnome to speak Finnish to me, even though the LANGUAGE and LANG settings are set the English. LC_TIME is something I'd like to use, but I find the Finnish abbreviations for weekdays and months to be confusing. LC_MESSAGES causes gnome to talk partially Finnish, the general locale is English as it's supposed to be, but for example gnome-panel changes the menu entries to Finnish.

I wish that it was easier to set these settings. I also doubt that various tools know how to respect these settings and will overwrite that file with the default setting. That is why I'm writing this entry so that I remember how to fix it when that happens.

In the end the system is flexible but it's built in a funny way. The actual variable built by combining 2 or 3 values: language, country and possibly encoding. So to make thing easy for me I would have to declare en_FI.UTF-8 locale and start translating applications to that locale. I don't want to, so I'm sticking to this "temporary" solution.

A while back wanted to know about a replacement for my Dreambox. I was also asked to write up a summary of the replies.

Initially I assumed that my Dreambox 7025C was failing, it showed a classic symptom, progressively increasing error rate which wasn't showing up on my neighbors. Since we share the cable they should see the problems too, so I ordered new tuners for the box and the problem persisted. In the end it turns out that the problems were initially showing on such channels that my neighbors weren't watching or in such amount that they didn't pay attention to it.

In the end it was a problem in the feed and my box is just fine. I now have 2 brand new DVB-C tuners for the 7025 (anyone want to buy them?) and a perfectly working setup. (YAY!).

In a way I feel lucky that the box wasn't failing on me. Nobody suggested any complete solutions, but I was able to spot ReelBox Avantgarde as a possible alternative. It runs Linux like I want it to, but it's way too expensive. Another alternative could have been Maximum 8000, which appears to be a re-branded Marusys C-8000. It has Linux as the OS but the community side appears to be rather lacking. Then again, it's a pretty new device...

I was suggested VDR and MythTV as the DIY solution. I dismissed MythTV without even looking in to it. Although I must admit that it's been quite a few years since I tried MythTV, but it was way too unstable for my taste. Also I wasn't too happy about the architecture of it.

VDR is something I could have tried. The downside appears to be that it lacks in hardware support. The last time I was building my own PVR I ran in to the common problem of getting a decent output to the TV. VDR folks have traditionally solved this by using a HW decoder card to output the stream directly to the TV. Using VDR with a budget DVB card is possible but not recommended because of the CPU power needed to decode the stream. I've always considered VDR to be the more sophisticated solution of the two, but it is starting to sound a lot more hassle than it's supposed to.

Also, the DIY solutions are hard to get in to a decent form. Usually the cases are bulky and custom modifications are required. I'm not one of those guys that install neon lights to their computers to make it look cool, but if I'm having it in my living room I expect it to look decent. Take a look at the VDR boxes section on the website for examples. While I admit that it's up to me to build a decent box, I'm not willing to spend too much time hand picking every single component so that it fits a casing and works with the given application.

Finally, I'd like to thank all the people that sent me comments. Even though I ended up sticking with the current setup, it was a valuable look in to what is out there. The Dreambox I have, doesn't have a HDMI output and lacks in few areas. Keeping that in mind I'm eventually going to take this same look in to the alternatives while picking a replacement for the current box. Lets hope VDR, MythTV and Elisa (to name a few) continue to evolve and the video output properties of Linux become better and eventually I'll have the possibility of replacing my current setup with something that is even more flexible.

For almost 2 years I've been a happy Dreambox user. While Dreambox has a lot of good features, the best being that it's based on Linux and people are actually encouraged to create their own firmwares and modify it, it lacks in hardware. All products suffer for some kind of disasters, for the 7025 series it was the power supply which will fail (it's only a matter of when). And now it looks like the rest of the hardware is failing too.

So, "Dear Lazyweb" I'm asking if there is something similar to Dreambox in terms of flexibility and features. I'm mostly looking for an active community and solid device. HD capability is a plus and it must have DVB-C receivers (2). I'm not looking to build my own box, but if there is a reasonable framework for that I will look in to it too. In general I'm looking for a box that is flexible and has solid hardware from the "(sane + a bit) < insane" price range.

If you happen to know of such a device or framework, drop me a note at ressu@ressukka.net (works with e-mail, jabber and msn, first 2 preferred =)

Sometimes I get the urge to vent a bit. Here goes...

It really annoys me to see websites that are designed in a stupid way. There are a lot of ways you can design a website and quite a few of them lead to problems that the designer didn't think about.

Designing websites isn't always easy. You need to be able to figure out all possible corner cases and prepare for them. Usually things go wrong right in the beginning. Developer picks a tool and decides that the tool is really nice and sticks with it. The thing is that the tool doesn't really make the website. It helps you to create one! A big part of creating a website is fine tuning the output of the tool.

The most common problem with web layouts is the orientation to the content. Most designers think that website should behave like a newspaper and have columns. Usually the right column has a menu and left column has ads or some other "relevant" information. The problem is that while doing this, designers usually opt for tables and images. Now, this presents us with a problem images fail to scale well and the table becomes static in width. What does the designer do? Opts for 800x600 or 1024x768 layout, depending on who they assume to be their target customers. This leads to horrible layouts, like the following.

While allowing the text to flow properly on the full window will get you some initial readability complaints, this is caused by the fact that people don't use 800x600 or even 1204x768 windows. Shocking, isn't it?! So we have web designers creating websites for window sizes that nobody actually uses.. What is the point in that?

Another thing that bugs be a lot is the use of JavaScript and cookies. Don't get me wrong, both of them are a good thing. There are a LOT of websites created with dynamic html which works really well. The problem is that too many sites rely on JavaScript and cookies when there is absolutely no need for either of them. Most of the common JavaScript tricks can be done with plain and simple CSS and too many sites use session to store data when there is no need to use a session. What makes things worse is the fact that hardly any of these sites check if cookies work or offer alternative navigation if JavaScript isn't available.

I just visited UPS which first presents me with a nice page about selecting my location. Since I use NoScript to block all scripts (yes, it makes the web a friendlier place) it alerts me about not being able to run scripts. Since there is very little on the page, we could assume that this is one of those "change the value of this drop-down and automatically submit"-scripts. Selecting the correct language and selecting submit brings me to a blank page.

Inspecting the front page a bit more reveals that the JavaScript actually changes the link to point to a completely different page. What this means is that if you ever visit the UPS front page with a browser that doesn't support JavaScript at all, you can't access the content at all.

In fact the whole selection is quite pointless. Yes, it brings me to the 'Finnish' page. The language is still English and there is hardly anything different from most other countries. Why was I forced to select my country? Why didn't the website guess? There are pretty accurate IP databases out there that can pinpoint the city you are accessing the site from. Language can be detected from the headers the browser gives your website. The whole front page is pointless and wastes my time.

In the end of this all ranting I must say that getting a website right doesn't happen by chance. It is possible with a lot of hard work and patience. And I appreciate those who manage big sites and get it right.

I was going to give more examples, but had a hard time finding them...

I just have to say it, memory management is utterly broken in Linux. Luckily it's not as broken as it used to be. Without going in to too much detail, there is a feature called overcommit in Linux. What this means is that it will over allocate memory to applications. At first thought it sounds insane. Why would one want to over allocate memory. It's like eating a cake and having it too.

Well, there is a reason why this is done. If you think about resource allocation in general, there are a lot of places where this is being done successfully. For example ISPs always over sell their capacity because most of the users use a fraction of the resources allocated to them. That is why guaranteed capacity is so much more expensive, they literally allocate it for you (most of the times). So in a way memory management is like your friendly sales person from Linux corp. who is happy to sell what over you want because he needs the deal for his own bonus.

So it is in fact reasonable to overcommit memory? Bzzt! Wrong! There are always problems with over allocating. ISPs, for example, monitor the usage of their resources and when certain limit is reached they get more resources. With memory it's not so easy, you have a finite amount of memory in your computer. Swap comes to the rescue to a certain degree, but it is critical for the applications to know their limits. If you have 1 MB of RAM available for the application let the application know. In most of the cases the application will crash because it doesn't know how to deal with the out of memory situation, but this is a side effect of the overcommit. Programmers don't run in to this kind of problems and they don't have to deal with them.

So what happens when you hit the limit? In Linux there is this thing called oom-killer. oom-killer has just one purpose, to kill processes. While oom-killer has some clever heuristics built in to it, it will most likely end up killing the wrong program. So it's like a big game of core wars on your computer.

Generating locales...
  en_AU.UTF-8... done
  en_BW.UTF-8... done
  en_CA.UTF-8... done
  en_DK.UTF-8... /usr/sbin/locale-gen: line 238:  1971 Killed                  localedef $no_archive --magic=$GLIBC_MAGIC -i $input -c -f $charset $locale_alias $locale
failed
  en_GB.UTF-8... done
  en_HK.UTF-8... done
  ...

This is what happens on one of my hosts, there are not that many services running on this host and for some reason the right process gets killed. After turning overcommit off:

Generating locales...
  en_AU.UTF-8... up-to-date
  en_BW.UTF-8... up-to-date
  en_CA.UTF-8... up-to-date
  en_DK.UTF-8... done
  en_GB.UTF-8... done
  en_HK.UTF-8... done
  ...

Apparently locale generation is one of the things that is capable handling memory as it's supposed to be handled.

So how do you turn off overcommit? Sysctl is the way! Easiest way is to add the following to /etc/sysctl.conf and running "/etc/init.d/procps.sh force-reload":

vm.overcommit_ratio = 100
vm.overcommit_memory = 2

The second line is the magic line which turns off overcommit. Insanely enough the other options are 0 (Default) and 1 (Always overcommit), which just doesn't make any sense. If the default is to do overcommit why would I want to set it to always overcommit. The first line defines the maximum amount of memory a program can allocate. This is explained better on a post on fedora-devel.

Even with these changes, overcommit isn't completely disabled, but things are a lot better. While overcommit is built because it makes sense, it doesn't work. Lets hope things get even better in the future.

Update: Arthur de Jong pointed out that in certain situations overcommit is needed, like when there is a large application running on a server that prevents further memory allocations (thus disabling remote access and so on). That is true, I'm not trying to say that overcommit isn't useful in some cases. I'm just saying that it's just a disaster waiting to happen, unless you have finetuned the environment just right.

Way back when I created my LVM array for the NSLU2 I opted for USB Stick that would hold the Swap and /var/log so that the actual disks could spin down. At the time I gave the scheme quite a bit of thought because I knew that the stick would eventually fail. That is why I decided to put the data which mattered the least on the long run. Of course failing stick would bring down the whole host because Swap was there, but it was an acceptable risk. It's not like the host needs to be online 99.999% of the time =)

Today the stick started presenting with errors and since it's already been a year since I started using the stick, it's most likely breaking down. A new stick costs next to nothing and I happen to have plenty of them laying around. Time to make the switch. The good thing about picking LVM for the disks is that the process is really simple. One would think.

I plugged in the new stick, created partitions on it and ran pvcreate and vgextend to add the disk to the volume group. Next it was time to move the data.

pvmove -v /dev/oldstick /dev/newstick

After this it should be a matter of vgreduce and we should be all set. Well, it's not as easy as that. Usually this would be the case since not too many use LVM mirroring. The thing is that pvmove doesn't know how to move mirrored drives. Which normally isn't a problem. You just reduce the mirror and extend it to the new drive. But in my case the mirror log is the one that is blocking my move. LVM handles mirrors by writing a log partition which requires a 3rd drive for a normal 2 drive mirror. There is a way around this, by keeping the log in the memory which I didn't want to because of the limited RAM.

LVM is capable of converting between mirrored and unmirrored logical volumes quite easily, but it's time consuming process to re-create the mirror. It's not like I'm in a hurry, but it still seemed too much. The solution is to convert the log in to an in-memory log, removing the old drive and converting the logs back to persistent disk logs.

lvconvert -m1 --corelog /dev/vg00/Vol1
vgreduce vg00 /dev/oldstick
lvconvert -m1 /dev/vg00/Vol1

It's pretty much the same amount of work to run that instead of removing the mirror and recreating it.

As for me, I have a working array again. The old stick has been dealt with (it's no longer operational and life is back to normal.

A while back my NSLU2 setup broke down, the reason was that an external USB drive failed. The HD model had a known problem that caused the drive to fail. The drive was under warranty at the time, but I knew that the actual HD was most likely working just fine. After a bit of pondering I decided to void the warranty and took out the HD from the case. After a few scares, the drive worked correctly.

When I bought that drive, I was smart and bought a second drive for backup purposes.. But as usual, the drive wasn't connected to anything and didn't have any backup value. So I had to set up the mirroring if I was to avoid future crashes, maybe I wouldn't be so lucky next time.

When I set up the software raid, I went with plain LVM mirroring. Almost all documents in the web talked about mdadm + LVM, but that would make my setup more complex and I didn't want to go that way with the limited CPU and RAM of the NSLU2. Setting up plain LVM mirroring was almost too easy, the only caveat was that for plain 2 drive mirroring you need 3 drives in the Volume Group. This wasn't a problem for me since I already had an USB Stick acting as the /var of the box (this allows the HDs to spin down).

So the complete process for adding a mirroring with LVM was just:

pvcreate /dev/sdc1
vgextend vg00 /dev/sdc1
lvconvert -m 1 vg00/Vol1

LVM will choose the drives it needs to use for the mirroring and start copying data. You can monitor the progress by doing

lvs -a -o +devices

That's all there is. Now, this post was supposed to be about recovering data from a broken volume group. A few days ago, the same drive took a dive from the table (maybe it figured out that it was on extended time already) and broke down (yes, we all know that clicking sound).

The NSLU2 hung and I quickly figured out that the only thing that wasn't mirrored was the data on /var and Swap. /var was on a USB Stick, but the Swap happened to be on that broken drive. OK, so kernel doesn't handle disappearing swap too well (at least if you have 32MB of ram). So I tried to reboot it, with little luck.

LVM isn't too helpful when it comes to broken drives. It will work just fine if you don't deactivate the Volume Group, but it refuses to activate it before it's fixed. I moved the working drives to my laptop and ran lvs to see the situation (udev already set up the VG for me)

lvs -P -a -o +devices

The -P flag means partial, so lvs includes the status of those Volume Groups which are missing physical volumes. The list verified that the Swap was actually on the broken drive but everything else was either mirrored on a working drive or on the stick. Removing the missing drives fixes the Volume Group and allows the drives to be mounted read/write.

vgreduce --test --removemissing vg00
vgreduce --removemissing vg00

The first run is just to see that it's actually capable of doing something sane. It will complain about missing drives on both run though.

Finally after the Volume Group was fixed I just ran fsck on all the volumes just to make it easier for the NSLU2 to boot. One of the partitions did have corruption, but I think it has been corrupted since the drive initially failed, it just wasn't scanned until now.

To remove the drives cleanly without reboot you need to run the following:

vgchange -a n vg00
sync

Then you can just unplug the drives.

Initially i made the mistake of exporting the VG before disconnecting. I wanted to make as clean exit as i could, but exporting the VG will change the state on disk and the NSLU2 will not boot before the VG is imported again.