Today while watching a Cisco talk about their Spam product, I realized that the current state of spam is never going to change. It's not because there isn't an effort to block spam, but because there is an effort to block spam.

At first thought it sounds odd, but it's the same problem as with pretty much everything. There is no money in fixing the current e-mail system, while there is a lot of money to be made in providing new solutions to block spam. Someone with a tendency to go with conspiracies would most likely claim that it would be in the best interest of some companies to fund sending spam.

Of course I'm not claiming this, but it would actually make sense.

But it's the same as with spam, spammers will be in the business just as long as there is money to be made from spamming. The companies working against spam are in the business just as long as there is spam going on. If spamming stops or someone develops a real solution to end all spam, the companies will go out of business or at least will have to look for other venues for income. And most companies don't like to invent new things, but rather sell the old product over and over.

I just hope that in the future the same companies will figure out that it's in everyones best interest to create a new end-all solution, even if it means that their current products will become obsolete. Instead they should focus on transition solutions as well as capitalizing on the fact that it's a lot of work to rework the current e-mail system.

It doesn't hurt to dream.

When Finland switched to Digital Television, I opted for a Dreambox. The box has been well worth the money. Almost a year ago (yes, I've been meaning to make a post about this ) I bought a Logitech Harmony One universal remote. Since the beginning it had some issues with the box.

The problem is that by default most of the commands repeated twice on the Dreambox. The remote itself can be modified to fit other devices, but it's not fine grained enough. You can basically alter the default setting of "2" to "1", what ever those mean. Keeping the setting at 2 gives duplicate presses and 1 gives me a poorly working remote (not all presses are registered).

Luckily the Dreambox is open and it's easy to dive in and dig around. After some digging around, I found /proc/stb/ir/rc/repeat which allowed me to alter the repeat time of the remote. After trial and error, I managed to find a working setting. Looks like 1150 ms is just a tad bit below the required repeat time for the remote, but for some reason increasing the setting to 1200 made things worse. So this appears to be as good as it gets for me.

I also created a script to set the repeat time at boot (since I shutdown the box for the night) which can be found here. Fix-repeat script takes install or uninstall as argument so it should be relatively easy to install it. The only real trick used in the script is the conversion from a decimal number to a HEX number.

After beating a dead horse once, you have to do it again.

Quite some time ago now, I ran in to an article where Wietse Venema was interviewed about security focused programming. He is the guy who initially started writing postfix. While he does have some good insights in to why the internet is getting more insecure, there are some things that I think he is missing.

While I understand that reinventing the internet is pointless, there are some things that need to be reinvented. The SMTP protocol is one of those things. The protocol itself was designed for a completely different type of a network. It doesn't do any authentication or verification for the sender. It basically trusts everything that is fed to it. Now, Wietse offers nice ideas on counteracting spam, but the inner problem still persists. The reason why we have so much spam in our mailboxes is because of SMTP.

I was about to write some code and put my money where my mouth is, but today I ran in to yet another scheme to stop spam and realized that I would eventually forget to write this all down before I get the time to write some code. The scheme I ran in to is EmailReg.org which allows you to register the mail servers that are allowed to send mail for your hosts for a nominal(?) fee of $20. While this nominal fee will keep some of the spammers off the lists, it's still reinventing the wheel and trying to fix a symptom of the problem.

There are similar projects, that work with different types of protections. There is SPF, DomainKeys, Sender ID and many more that aim for similar solutions.

I'm not saying that we don't need those kinds of solutions, but rather the solutions are solving the symptom and not fixing the problem. The only real solution is to replace SMTP with something more suitable for the task. The thing that comes in to mind is XMPP. XMPP is a protocol designed for XML packet routing. It's mostly used as an instant messaging platform, but it's not a huge leap to transfer mail through the protocol.

XMPP as a protocol is designed on an age when spoofing and spamming was already a problem and it has safeguards in place to prevent malicious activity. The protocol is suitable for transferring e-mails already so no real modifications are needed for the protocol. Only thing that needs to be done is to document the common practice.

Changing an internet protocol is a large task and one can't take it lightly. The beauty in it all is that most mail servers are already capable of supporting multiple transport protocols. Initial versions of Sendmail delivered the mail through FTP (with some obscure extensions), so switching protocols isn't really that far fetched. Today, Sendmail supports various protocols while the most commonly used one is SMTP. Same goes for postfix and other mail servers. So implementing a new protocol isn't really out of the question.

I'm sorry that I wasn't able to write the code to back this all up, but at least the idea is out there in written form. It shouldn't be too complicated to implement this and get things started. The change won't happen over night, but it has to start somewhere.

It's no wonder people hate dealing with certificates. I'm one of those people who really hate handling certificates. For some reason Linux lacks simple tools to manage certificates. Debian has one set of tools and I bet quite a few other distributions have their own set of tools. So nothing generic.

In my opinion the problem lies in OpenSSL, which is an overly complicated piece of software. Sure it's able to encrypt your fries at the local fast food place, but most people never use anything more than the x509 module. And even that is complicated.

I'm not saying that it should be point and click operation, but what I want is a tool that allows me to create, renew and verify SSL certificates. Which is pretty much the most common thing you do with OpenSSL.

Now, lets assume you have a certificate and a key that you want to check if they match. Can you say from the top of your head how to do that? If so, you are either dealing with this stuff daily or you looked it up from a manual. The correct "magical mumbo jumbo" is:

server:/tmp# openssl rsa -noout -modulus -in /etc/ssl/private/some.key | openssl md5
94c5fa8b0fe1baa6e8dfcdec75444620
server:/tmp# openssl x509 -noout -modulus -in signed.crt | openssl md5
94c5fa8b0fe1baa6e8dfcdec75444620

Now, how easy was that. It only took me 10 minutes to construct that line. Most of the time went to searching Zimbra scripts for the correct magical line. The reason why I went through the scripts instead of the manual is that I had already seen the scripts do a appropriate check. And there is no way I could have constructed that in that kind of a time frame just by using the manual.

Usually I think that people complain too much when they can't figure out how to make ls or something work like they want. A certain degree of manual reading is good for you, but in this case it's too much. In any case, I'm posting this rant as a reminder for myself so that the next time I can just look it up from here.

Sometimes I wonder why I'm not a millionaire... I keep having these odd strokes of luck.

Today the HD on my laptop broke. And I had such a good plan for backups. It should have been foolproof. I meant to run incremental backups every so often and put the data in to external drives, which are mirrored. It turns out there was a flaw in my backup plan: I forgot to implement it. Yes, my HD broke down and I had absolutely no backups of my HD contents. Since it's not the first time I've seen a HD failure, I immediately started making a clone of the disk. I didn't want to waste time looking for individual files, since a clone of the disk contains all the data that is accessible on the broken disk anyway.

I decided to try with clonezilla, because it's intelligent enough to backup the data I need but still has a decent GUI to work through the command line options. After all, I don't clone disks daily.

After starting the clone, it tried to access the data and after a while I saw a message from kernel saying that the disk froze. I almost gave up hope when the partitions showed up on screen. I gasped for air when I finally realized what happened. Apparently the swap partition was the only thing that was lost! Feeling hopeful I left clonezilla to do it's job and headed out the door. Once I got back all of the data had been backed up.

At this point I already had a new drive so it was time to do the reverse and restore the backup, which was an uneventful 2 hours of waiting. Before rebooting I ran mkswap on the swap partition and rebooted.

The system is now back up and running.

It's not the first time I had problems with HDs, recently I've lost 3-4 HDs due to various failures. Only once I've lost data, but the lost data wasn't anything that couldn't be regenerated. Maybe it's not luck, maybe I'm a cat in a human form. Maybe I just always land on my feet... Time will tell.

Recently I fixed my OpenVPN tunnel and figured out what was wrong with my bridge setup. Here is the complete setup that I have.

Server

I wanted to have as much of the configuration on the server as possible so that I could easily add more clients and wouldn't have a need to update client configuration when ever the server preferences change.

Here is my OpenVPN server configuration:

mode server
tls-server
dev tap0
server-bridge 10.5.6.1 255.255.255.0 10.5.6.200 10.5.6.209
ifconfig-pool-persist /var/run/openvpn-ip.txt
keepalive 10 60
ping-timer-rem
persist-tun
push "route 10.5.6.0 255.255.255.0"
push "dhcp-option DNS 10.5.6.1"
ca /root/easy-rsa-2.0/keys/ca.crt
key /root/easy-rsa-2.0/keys/ressukka.net.key
cert /root/easy-rsa-2.0/keys/ressukka.net.crt
dh /root/easy-rsa-2.0/keys/dh1024.pem
client-to-client
up /etc/openvpn/ovpn-ressukka.sh

and the up script:

#!/bin/sh
# Bind the tunnel interface to the bridge

BRIDGE=br0

ifup $1
brctl addif $BRIDGE $1

There is nothing really special about that configuration. The server is in TLS mode configures a bridge. The keys are generated with easy-rsa by following a openvpn howto entry. The up script just binds the tap0 device to the network bridge after bringing up the device.

Next I created the interface configuration by adding the following to /etc/network/interfaces:

iface br0 inet static  
 address 10.5.6.17  
 gateway 10.5.6.1  
 netmask 255.255.255.0  
 bridge-ports eth0

The trick here is to create a single bridge with just the eth0 device. We use the up script for openvpn to add the tunnel device to the bridge. Otherwise the bridge would never contain the proper devices.

Client

As for the client, you simply set the client to use the CA-certificate and Host key created with easy-rsa, set the hostname and tunnel type. Tunnel type is assumed to be a tun-device instead of tap, so in my case I needed to change it too.

There is no need to tell the client anything else. Everything else will be negotiated through the tunnel. And since I use NetworkManager to set up my tunnels I didn't have a need to drop in to a shell even once at the client.

I've been meaning to write up on this for quite some time now, but always have something seemingly more important things to do.

Most modern Linux distributions offer ways to manage network interfaces via some kind of abstraction. Usually this abstraction allows one to dynamically rename and add network interfaces. For Debian family management is done with ifupdown, while with RedHat family it's sysconfig. In addition there is NetworkManager which is a cross platform solution for dynamic network configuration (which eliminates the need to rename interfaces).

Case Xenserver

Usually this comes in rather handy, but on other occasions it can be a pain. A while I was helping out with an installation of a XenServer instance. This server had, for some reason, ethernet interfaces reversed in comparison to the other XenServers on the site. Luckily the service console has the network configuration in ifcfg scripts. We were easily able to reverse the interfaces by binding the interfaces to certain hardware addresses.

The only problem is that the interface is renamed only when the interface is brought up. What's worse the interfaces were enumerated before the server would bring up all of the interfaces. Only eth0 was brought up for management purposes before enumerating. This means that the original eth0 was renamed to _tmp_xxxxxx. The (oh so elegant) solution was to create a script that does ifup eth1; ifdown eth1. Problem solved.

Case tap0

I also experienced similar problems when I was setting up my OpenVPN tunnel. I wanted to use a bridged connection to my network, but for that I needed to create a br0 interface with tap0 as a member.

It's easy enough to create ifupdown configuration to set up tap0. Hook that up to br0 and you are all set. The same problem will bite you here. The interface tap0 is actually created only when the tap0 is brought up and br0 members are added only if they are present.

The solution? In my case, create a custom script that adds the device manually after it's been created properly. I was unable to find anything that works without using a custom script =(

In retrospect

Looking back at these cases, I should have known better. The problem was obvious and it took me way too long to figure out the cause for my problems. Then again, the utilities should be able to create a reasonable abstraction for themselves that they use to determine the actual status of the whole system. This way the trickery needed for setting up simple interface would be obsolete.

You can't have it all, but you can always hope.

After my post yesterday about locales and my problems, I got some comments.

Bryan, Simon and Wouter all pointed out what eventually figured out too but didn't mention in the post (I actually had to re-read the post to see what I wrote) that LC_ALL overrides all of the settings. Even though I figured it out eventually, with these comments it finally me that the purpose is actually to temporarily override the locale.

Another thing that didn't occur to me while figuring out the correct locale for my system was that I'm thinking about it all wrong. Since I've already gotten used to broken locales I kept thinking that I want en_US locale with some Finnish settings, but in reality I wanted fi_FI locale with English language. I have to admit that I was pretty sceptical about the solution but decided to try it out. And to my surprise, it actually worked!

So the final solution is this: LANG="fi_FI.UTF-8"
LANGUAGE="en"
LC_MESSAGES="en_US.UTF-8"
LC_TIME="en_US.UTF-8"

It is a pretty clean solution and I'm happy to live with it. And as Simon pointed out, administrator changes should be preserved through upgrades. If not, one should file a bug for it.

There we go, another problem solved. Thanks for the comments and suggestions.

Linux has this great thing called locales. You can basically control everything through a few system variables. The system is so flexible that you could in fact have English language with Swedish dates and Chinese error messages.

The problem with the current system is that it's controlled mostly with one variable. Just about everyone I know (including me) set the system locale to en_US because they don't want to use their local language as the system language. This in turn causes new problems, mostly due to the metric system used in the sane countries.

I finally got around to debug how to get the system to speak my language. The key here is the file /etc/default/locale which on Debian based systems contain the locale settings. Usually there is just one line in that file.

LANG=en_US.UTF-8

Since I live in Finland and want my computer to speak English I can add the following lines:

LC_NUMERIC="fi_FI.UTF-8"
LC_PAPER="fi_FI.UTF-8"
LC_NAME="fi_FI.UTF-8"
LC_ADDRESS="fi_FI.UTF-8"
LC_TELEPHONE="fi_FI.UTF-8"
LC_MEASUREMENT="fi_FI.UTF-8"

This replaces the settings for numerals (decimal separator and such), paper (yes, the rest of the world uses A4), name, address, telephone and measurement (YAY, metric system!). This way I have a nice English speaking system with Finnish settings.

I tried setting LC_ALL to fi_FI.UTF-8 but that causes gnome to speak Finnish to me, even though the LANGUAGE and LANG settings are set the English. LC_TIME is something I'd like to use, but I find the Finnish abbreviations for weekdays and months to be confusing. LC_MESSAGES causes gnome to talk partially Finnish, the general locale is English as it's supposed to be, but for example gnome-panel changes the menu entries to Finnish.

I wish that it was easier to set these settings. I also doubt that various tools know how to respect these settings and will overwrite that file with the default setting. That is why I'm writing this entry so that I remember how to fix it when that happens.

In the end the system is flexible but it's built in a funny way. The actual variable built by combining 2 or 3 values: language, country and possibly encoding. So to make thing easy for me I would have to declare en_FI.UTF-8 locale and start translating applications to that locale. I don't want to, so I'm sticking to this "temporary" solution.

A while back wanted to know about a replacement for my Dreambox. I was also asked to write up a summary of the replies.

Initially I assumed that my Dreambox 7025C was failing, it showed a classic symptom, progressively increasing error rate which wasn't showing up on my neighbors. Since we share the cable they should see the problems too, so I ordered new tuners for the box and the problem persisted. In the end it turns out that the problems were initially showing on such channels that my neighbors weren't watching or in such amount that they didn't pay attention to it.

In the end it was a problem in the feed and my box is just fine. I now have 2 brand new DVB-C tuners for the 7025 (anyone want to buy them?) and a perfectly working setup. (YAY!).

In a way I feel lucky that the box wasn't failing on me. Nobody suggested any complete solutions, but I was able to spot ReelBox Avantgarde as a possible alternative. It runs Linux like I want it to, but it's way too expensive. Another alternative could have been Maximum 8000, which appears to be a re-branded Marusys C-8000. It has Linux as the OS but the community side appears to be rather lacking. Then again, it's a pretty new device...

I was suggested VDR and MythTV as the DIY solution. I dismissed MythTV without even looking in to it. Although I must admit that it's been quite a few years since I tried MythTV, but it was way too unstable for my taste. Also I wasn't too happy about the architecture of it.

VDR is something I could have tried. The downside appears to be that it lacks in hardware support. The last time I was building my own PVR I ran in to the common problem of getting a decent output to the TV. VDR folks have traditionally solved this by using a HW decoder card to output the stream directly to the TV. Using VDR with a budget DVB card is possible but not recommended because of the CPU power needed to decode the stream. I've always considered VDR to be the more sophisticated solution of the two, but it is starting to sound a lot more hassle than it's supposed to.

Also, the DIY solutions are hard to get in to a decent form. Usually the cases are bulky and custom modifications are required. I'm not one of those guys that install neon lights to their computers to make it look cool, but if I'm having it in my living room I expect it to look decent. Take a look at the VDR boxes section on the website for examples. While I admit that it's up to me to build a decent box, I'm not willing to spend too much time hand picking every single component so that it fits a casing and works with the given application.

Finally, I'd like to thank all the people that sent me comments. Even though I ended up sticking with the current setup, it was a valuable look in to what is out there. The Dreambox I have, doesn't have a HDMI output and lacks in few areas. Keeping that in mind I'm eventually going to take this same look in to the alternatives while picking a replacement for the current box. Lets hope VDR, MythTV and Elisa (to name a few) continue to evolve and the video output properties of Linux become better and eventually I'll have the possibility of replacing my current setup with something that is even more flexible.