This is an ongoing set of benchmarks against various Unix (Linux, Solaris, MacoSX etc.) distributions, trying to get a feel for how they compare on CPU performance. This is by no means a complete test, I'm only running a few benchmarks on each distribution, and I'm primarily only looking at raw CPU performance. Also, in order to make this reasonably easy for me to handle, all tests are done inside a VirtualBox virtual machine, and only one CPU is ever used (meaning, the tests don't show how well the Unix flavor manages SMP.

The test system is a Linux FC9 box, with a Core2 6600 CPU running at 2.40GHz, with 4MB of cache. Each Unix distribution is installed in a VirtualBox (v2.10) virtual host, with 512MB of RAM (plenty for all tests). All Linux distributions were updated with all the latest patches available at the time of the test, running whatever kernel, compilers and libraries available at the time.

Each benchmark is run three times on each distribution, and the best result is picked from each test. Currently the only benchmark I run is the ByteMark, called nbench. The source for this is available at http://www.tux.org/~mayer/linux/bmark.html . As mentioned before, this not only benchmarks the kernel, it also exercises the compiler suite, and supporting libraries (e.g. glibc). But then again, someone picking a distribution probably wants to see the "whole picture", right?

I'm planning on running lmbench on these machines as soon as possible, to see if those numbers shows any more significant differences. I'm also planning on adding benchmark results for SUSE, which is a popular Linux distribution. Unfortunately I can't test MacOSX on this hardware, since I can't install it under VirtualBox. In particular, lmbench would probably be more useful without Virtualization, for testing how well the system behaves on things like disk I/O, network and SMP scalability.

With time permitting, and if/when I can free up the hardware, I think it'd be useful to do these benchmarks without virtualization involved. It's unclear today what impact VirtualBox has on these benchmarks (but, I'm hoping it's relatively fair across all the distributions). When Michelle lets me upgrade my desktop, I'll use the old system for rerunning these tests.

While working on some completely unrelated piece of code, I discovered what seemed to be the impossible: FreeBSD was much faster than my 2.6 Linux system on certain system calls. And it was quite noticeable, a magnitude or more. This clearly couldn't be right, so I ended up doing some tests and I wrote a blog entry on it. Well, this continue being an issue, so I decided to expand on the topic.

The primary test system is a Pentium 4 2.8GHz with HT, so all kernels are built with SMP support. For comparison, I've run the same tests on a few other platforms, including FreeBSD. I don't claim any of these tests to be scientifically correct in any way, but something is definitely wrong here.

I checked the same code on my older system, running a 2.4 kernel on an old Celeron CPU. To my satisfaction, it was indeed very fast, even faster than FreeBSD. I double checked my PIII machine running a 2.6.6 kernel, it was also reasonably fast.

So, why was my main Linux system so damn slow? Well, as it turns out, not all kernels are created (i.e. built) equal/ I did a number of benchmarks on my system, and discovered that pretty much all my pre-built kernels (from Fedora Core 2 "rawhide") were slow, but only on a Pentium-4 system. My custom built kernel was fast

The numbers below are clock cycles per system call (smaller is better):

Kernel	gettimeofday()	uname()	chdir()	open()
FreeBSD 4.x on P4	1591	29446	8977	2583
RedHat 2.4 on Celeron	461	592	1198	864
FC2 2.6.6 on PIII	1076	1286	2547	1425
FC2 2.6.3 i686 8k stack on P4	8373	1681	33115	29725
FC2 2.6.7 i686 4k stack on P4	8453	8680	41695	37885
FC3 2.6.9 i686 on P4	8720	9058	14154	9887
custom 2.6.7 8k stack on P4	890	1031	4601	1396
custom 2.6.7 4k stack on P4	828	1002	4295	1377
custom 2.6.9 on P4	814	990	4514	1315

My custom kernel configuration is "optimized" for my particular system, the configs are available here. The simple program that I used to benchmark this is available here. This code only works on x86 platforms.

First of all, this is obviously only a problem with P4 systems, so I did a bit research (but not a lot). Apperently P4 CPUs have a new instructions for handling system calls, SYSENTER, while older Pentium systems uses the 0x80 interupt. As far as I can tell, there's a new virtual system call layer that is supposed to handle this, called vsyscall. There's some info on this in this Kerneltrap.org article. I guess I assumed that vsyscall would handle all combinations of CPUs and builds, but maybe it's not?

The solution for me was to recompile the "stock" Linux kernel, based on the FC3 kernel configuration file. I did quite a few changes to the config, and I also didn't apply any of the RedHat/Fedora patches to the kernel source.

The latter is the key to the solution, after trying numerous configurations and options, I narrowed it down to the set of patches Fedora applies. Dave Jones kindly confirmed this, and pointed to the exact set of patches: exec-shield. If you kernel applies the same or a similar patch, you should definitely be aware of this problem.

I haven't yet found an easy way to build the Fedora kernel without just the exec-shield patches, but I'm still working on that. Unfortunately the patches co-depend on each other a bit, so it's not as simple as just leaving out the patch.

I've filed a bug against Fedora Core for this, see Bugzilla bug #139318.

Obviously system call performance isn't a huge factor on overall system performance, although I can see certain applications suffering more than others due to this problem. To get a somewhat interesting comparison, I did a compile of the kernel source on a "slow" kernel vs my custom built one. Here are the results, output is from the time command:

"Slow" kernel:

# time gmake
real    21m10.495s
user    19m0.013s
sys     3m6.801s

"Fast" kernel:

# time gmake
real    19m17.596s
user    17m55.808s
sys     1m51.115s