The reason Linux systems lock up like that is because the kernel will let a process fill all memory up with dirty pages that need writeback, then as soon as it needs some memory the first thing it does it drops all of the in-memory copies of file-backed pages, which includes all of your programs. Then whenever one of your programs wants to run, or continue running by branching to a far address, it has to swap in that code from disk again. Even though you thought your system does not "have swap", it does have swap in effect. The workaround for this is to copy important programs into memory and pin them there with mlock. It is particularly important that if you rely on a userspace OOM killer it gets locked into memory.
I've also found that under these conditions kswapd will effectively consume all your CPU time. The time it spends running is probably proportional to your maximum memory too - in our case it parses through 500+GB of LRU. The blocking writeback behaviour can be managed effectively with dirty page writeback ratio tuning. You don't want to block trying to write 50GB to disk at once when you hit the high dirty page watermark.
The amount of dirty pages the kernel keeps in RAM is configurable through sysctl. If the buffer is full any further write blocks the process. If you have less free RAM than the allowed dirty page buffer what you said is correct though.
There is a new patch[1] that allows to set a soft and hard minimum of RAM reserved for clean pages. This fixes the problem almost completely even under the heaviest loads.
cgroups like a silbling post mentioned can also help by setting soft limits for heavy background tasks like compile jobs. Setting a soft limit will give them as much RAM as is available, or swap them out completely when things are heavily contented, effectively pausing the processes. It requires some setting up though, so it's not a solution for all cases, but it can make sense even without the thrashing problem.
> The workaround for this is to copy important programs into memory and pin them there with mlock.
Nit: I don't think it's necessary to copy the program into anonymous memory to use mlock. You might be thinking of huge pages (transparent or otherwise), which are supported on anonymous pages and unfortunately aren't yet supported on ext4- or btrfs-backed file pages.
The resource control aspects of cgroups actually improve this situation, if you take advantage of them.
You'll still see that page fault thrashing but it becomes isolated to the cgroup experiencing the memory pressure. It doesn't bring down the entire system in my experience.
I'd never considered the program itself getting dropped and re-read from disk -- but it makes perfect sense. I'm curious if executing the programs from a ramdisk would achieve the same effect.
