I rarely bother to "rate limit" the amount of CPU my Linux processes can do, but since I unfortunately have to share my desktop with some CPU intensive tasks, sometimes it makes sense to do so, for multi-threaded applications at least. Linux has, for some time now, a feature to set CPU affinity for a process. This is btw something that Sun/Solaris have had for almost as long as I can remember. With this feature, you can specify which CPUs (cores) a process is allowed to use.

As an example, lets assume we want the application traffic_server to only consume core 2 and 3, you'd simply start the application with

# taskset -c 2,3 traffic_server

Now, some applications have this built in (unfortuantely traffic_server does not, yet), and there are APIs in linux to control this (see the man page for sched_setaffinity() ). You can of course also "nice" your application accordingly, but the above lets you get a somewhat more controlled behavior (again, for threaded apps that can use more than one core).

To add to the pool of braindead benchmarks, but perhaps with a little more reason, I'm adding this, and take it for what it is. If anything, this shows that performance is generally not the primary argument for choosing an intermediary. This is what I've been preaching - Yeah, performance is important, but most servers available today will handle a ridicious amount of HTTP traffic.

This is a test against an AMD Phenom(tm) II X4 940 Processor (very cheap), running across a GigE network. There are two linksys switches between the load geneating host and the server, but no routing or packet filtering. The payload is 100 bytes + a fairly small header, and the test is running with keep-alive.

In all tests, all logging is disabled.

Varnish

The configs are mostly the defaults, the main thing was I had to jack up the minimum threads, 200 seems to be a reasonable number for this test. During the test, the load goes to 300. The version of varnish is v2.1.5, from the Fedora repository.

5719306 fetches on 450 conns, 450 max parallel, 5.776500E+08 bytes, in 60 seconds
101 mean bytes/fetch
95320.7 fetches/sec, 9.627390E+06 bytes/sec
msecs/connect: 3.955 mean, 6.453 max, 0.162 min
msecs/first-response: 3.546 mean, 1005.235 max, 0.076 min

Nginx

This is running the older v0.8.53 version, since it's what was made available on the Fedora repo. The configs had to be tuned some, increasing the number of worker processes, setting the open_file_cache high, and also increasing the keepalive_requests setting (high).

5848823 fetches on 450 conns, 450 max parallel, 5.848820E+08 bytes, in 60 seconds
100 mean bytes/fetch
97480.4 fetches/sec, 9.748040E+06 bytes/sec
msecs/connect: 1.340 mean, 3.558 max, 0.469 min
msecs/first-response: 3.522 mean, 280.463 max, 0.067 min

Apache Traffic Server

This is the winner, of course, otherwise I wouldn't have published these results ;). This is running ATS v2.1.8, with mostly stock config. The primary configuration changes is to set the number of worker threads to 5 and turning off some verbose Via and server strings.

6944993 fetches on 450 conns, 450 max parallel, 6.945000E+08 bytes, in 60 seconds
100 mean bytes/fetch
115748.6 fetches/sec, 1.157486E+07 bytes/sec
msecs/connect: 1.805 mean, 2.995 max, 0.519 min
msecs/first-response: 1.736 mean, 218.573 max, 0.081 min

Update: I updated with the latest results from ATS v2.1.9, they are marginally different.

To be proactive, I've found that once in a while (perhaps via a cron job) it might be a good idea to force a Linux RAID (mirror or RAID5/6) to be checked for consistency. This can easily be done from command line, with something like

% sudo echo check >  /sys/block/md0/md/sync_action

Also, to repair bad raid device, perhaps something like

% sudo echo repair >/sys/block/md0/md/sync_action

This second command solved a problem for me, where I'd get an email warnings once a week, saying "WARNING: mismatch_cnt is not 0 on /dev/md0". This was verified with

% sudo cat /sys/block/md1/md/mismatch_cnt
256

While testing and benchmarking some new features in Apache Traffic Server on my dev box, I started having really odd problems with inconsistently lost connections or really slow performance. Upon examing the logs, I found a large number of kernel barf like

Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.
Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.
Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.
Oct 29 11:53:51 loki kernel: nf_conntrack: table full, dropping packet.

Clearly this could not be good. I poked around, and could not find any ways to turn off this kernel module, at least on my Fedora Core 13 it seems compiled straight into the kernel (i.e. no .ko). There are ways to increase the table size (sysctl's) but that seemed like a band aid at best. Bummer. So, I went to the mountain (Noah F.) and asked for advice. After some poking around, we found that forcing iptables to ignore the conntracking helps, a lot. E.g.

iptables -t raw -I OUTPUT -j NOTRACK

 seems to do the trick for my particular case. This turns it off for all protocols, but you could probably limit it further (e.g. for tcp only with a "-p tcp" option). Also, you might want to do this on the input as well, e.g.

iptables -t raw -I PREROUTING -j NOTRACK

We found this old report against Fedora Core 10 as well, which describes the same problem, but no solution.

Apparently, when doing "yum update", and it fails miserably, you can end up with duplicate versions of packages in the RPM database. This seems harmless, but is annoying. yum provides a tool to check for this, but I was not able to find anything that would automatically repair it. So here's a little tip:

$ yum check duplicates | awk '/is a duplicate/ {print $6}' > /tmp/DUPES
$ yum remove `cat /tmp/DUPES`

 Of course, before you remove the dupes, make sure to examine the tmp file (/tmp/DUPES) and make sure it looks ok.

Update:

There seems to be a command to do this, package-cleanup has an option for it. E.g.

$ package-cleanup --cleandupes

However, testing this command on a second box having the same problem gave bad results, it seems to have uninstalled the "real" packages too.