I just have to say it, memory management is utterly broken in Linux. Luckily it's not as broken as it used to be. Without going in to too much detail, there is a feature called overcommit in Linux. What this means is that it will over allocate memory to applications. At first thought it sounds insane. Why would one want to over allocate memory. It's like eating a cake and having it too.
Well, there is a reason why this is done. If you think about resource allocation in general, there are a lot of places where this is being done successfully. For example ISPs always over sell their capacity because most of the users use a fraction of the resources allocated to them. That is why guaranteed capacity is so much more expensive, they literally allocate it for you (most of the times). So in a way memory management is like your friendly sales person from Linux corp. who is happy to sell what over you want because he needs the deal for his own bonus.
So it is in fact reasonable to overcommit memory? Bzzt! Wrong! There are always problems with over allocating. ISPs, for example, monitor the usage of their resources and when certain limit is reached they get more resources. With memory it's not so easy, you have a finite amount of memory in your computer. Swap comes to the rescue to a certain degree, but it is critical for the applications to know their limits. If you have 1 MB of RAM available for the application let the application know. In most of the cases the application will crash because it doesn't know how to deal with the out of memory situation, but this is a side effect of the overcommit. Programmers don't run in to this kind of problems and they don't have to deal with them.
So what happens when you hit the limit? In Linux there is this thing called oom-killer. oom-killer has just one purpose, to kill processes. While oom-killer has some clever heuristics built in to it, it will most likely end up killing the wrong program. So it's like a big game of core wars on your computer.
Generating locales... en_AU.UTF-8... done en_BW.UTF-8... done en_CA.UTF-8... done en_DK.UTF-8... /usr/sbin/locale-gen: line 238: 1971 Killed localedef $no_archive --magic=$GLIBC_MAGIC -i $input -c -f $charset $locale_alias $locale failed en_GB.UTF-8... done en_HK.UTF-8... done ...
This is what happens on one of my hosts, there are not that many services running on this host and for some reason the right process gets killed. After turning overcommit off:
Generating locales... en_AU.UTF-8... up-to-date en_BW.UTF-8... up-to-date en_CA.UTF-8... up-to-date en_DK.UTF-8... done en_GB.UTF-8... done en_HK.UTF-8... done ...
Apparently locale generation is one of the things that is capable handling memory as it's supposed to be handled.
So how do you turn off overcommit? Sysctl is the way! Easiest way is to add the following to /etc/sysctl.conf and running "/etc/init.d/procps.sh force-reload":
vm.overcommit_ratio = 100 vm.overcommit_memory = 2
The second line is the magic line which turns off overcommit. Insanely enough the other options are 0 (Default) and 1 (Always overcommit), which just doesn't make any sense. If the default is to do overcommit why would I want to set it to always overcommit. The first line defines the maximum amount of memory a program can allocate. This is explained better on a post on fedora-devel.
Even with these changes, overcommit isn't completely disabled, but things are a lot better. While overcommit is built because it makes sense, it doesn't work. Lets hope things get even better in the future.
Update: Arthur de Jong pointed out that in certain situations overcommit is needed, like when there is a large application running on a server that prevents further memory allocations (thus disabling remote access and so on). That is true, I'm not trying to say that overcommit isn't useful in some cases. I'm just saying that it's just a disaster waiting to happen, unless you have finetuned the environment just right.