leif's blog

Moab 2011: Day 2

With Randy's bike being basically in pieces, and not having the right tools to get it fixed, we decided to drop it off at the local Yamaha dealer here in Moab. They were great, took in the bike immediately, and promised to have it done today. After that, and the mandatory Egg McMuffin, we went and rode Fins and Things and Porcupine Trail. I of course forgot the battery for the Lumix camera that I brought, so no good pictures of the great views, but here's one shot taken with the video camera:

FILE0040

We rode around for a couple of hours, and got a solid ~21 miles on the bikes again. In the afternoon, Ted (Randy's dad) arrived, and we all hung out waiting for Randy's bike to be fixed. We picked it up around 5.30pm, and went out for another 7 miles of "test riding" it, and viola, we have three bikes again! I've uploaded a few more movies to my Contour account, here are a few links again:

http://contour.com/stories/moab-day-2-1

http://contour.com/stories/moab-day-2-2

http://contour.com/stories/moab-day-2-3

http://contour.com/stories/moab-day-2-4

Looking forward to a good, long day of riding tomorrow, with no broken bikes (or bones).

Misc: 

Moab 2011: Day 1

This is obviously not the usual techno babble on this blog: This week I'm in Moab (Utah) with my buddy Randy riding our dirt bikes.

P1000193

Unfortunately we've started off the week with a pretty rough start. Randy's bike is running like crap, and he ended up having to ride his dad's bike today. Even so, we got a nice ~22 miles on the bikes at and near Gemini Bridges. This is a beautiful place, and the "bridge" is pretty darn cool.

P1000202 P1000194

I've uploaded a couple of the videos: http://contour.com/stories/moab-day-1 and http://contour.com/stories/moab-day-1-2 .

I'm uploading a few pictures to Flickr as well, which I'll make available here soon. Tomorrow we'll hopefully head up to Porcupine rim in the morning, and perhaps Fins and Things in the afternoon. Another update tomorrow as well, stay tuned.

Misc: 

Fedora Core 15 and NIC devices ...

I've been upgrading our internal file server at home, and I've done that by rebuilding it in a separate box, and later moving the new drives over to the real server machine. This is all great. However, when I decided to do the final migration, I edited the IP on the "development" box to be the real server IP, but doing so instantly changes the IP on the machine. And therefore, I took out the old (in-use) server ... This seems like an incredibly bad idea, particularly for a server, but for any networking in general.

These changes should clearly not take effect until I either reboot, or restart the network services. What were they thinking?? Also, while I'm griping, why are there two sets of Firewall 'startup' and configs in Fedora Core 15? It's not enough to simply turn off the iptables service any more, you also have run system-config-firewall and turn it off from there (or manually edit its config file I guess).

I tip for the Fedora Core developers: FC (and RHEL) has always been about the server, please continue to make that the priority. All this focus on the desktop will only make you lose market share, Ubuntu alread does a good job there. At a minimum, if I cared about things such as Gnome 3, and automatic "firewalls" and what not, make it a special Fedora spin.

Filtering Drupal comment spam

I get a fair amount of comment spam on my blog, and even after I changed all comments to be moderated, the spammers still persist. I decided to do something about this, and working under the assumption that most spammers are from a few countries, I decided to implement a Geo-location filter for Apache Traffic Server. The code is currently available at http://svn.apache.org/repos/asf/trafficserver/plugins/geoip_acl/, and only works with MaxMind's APIs (but I'd be more than happy to add support for other Geo-location APIs). This plugin also requires PCRE, but that's already a requirement for building ATS, so shouldn't be a problem.

Once compiled and installed (see the README), setting this up is fairly straight forward. In my remap.config, I now have the following rule

map http://www.ogre.com http://localhost:69 @plugin=geoip_acl.so @pparam=country \
       @pparam=regex::/home/server/etc/deny_spam.conf


This says to apply a country based Geo-location filter on this rule, using the additional configurations from deny_spam.conf. This file contains one single line:

^comment/       deny    CN RU IN

This might look draconian, but for now I'm disabling all comment posts from China, Russia and India. For more details on the plugin configurations and features, again see the README from the source above.

Enjoy!

Hacking: 

yum failures with missing $releasever

During an upgrade (yum update) on a Fedora VM, something went horribly wrong, and it crashed in the middle of the update. After rebooting, and cleaning up the mess, yum still was very unhappy. Running an update would give me errors like

Could not parse metalink https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=x86_64 error was 
No repomd file
Error: Cannot retrieve repository metadata (repomd.xml) for repository: fedora. Please verify its path and try again

Very odd. It turns out, $releasever was not properly set, and I could not figure out why. Poking around, I realized that $reelasever is supposed to come from examining the version number of a particular RPM package, in my case fedora-release. Well, lo and behold, this package was no longer installed on my box, yum must have uninstalled it, but crashed before installing the new update (or something...). I mounted the Fedora Core DVD, and simple reinstalled the missing package, and things are happy joy joy again. Here's the command:

$ sudo rpm -i ./Packages/fedora-release-13-1.noarch.rpm

 

Hacking: 

Limit a threaded Linux app's CPU consumption

I rarely bother to "rate limit" the amount of CPU my Linux processes can do, but since I unfortunately have to share my desktop with some CPU intensive tasks, sometimes it makes sense to do so, for multi-threaded applications at least. Linux has, for some time now, a feature to set CPU affinity for a process. This is btw something that Sun/Solaris have had for almost as long as I can remember. With this feature, you can specify which CPUs (cores) a process is allowed to use.

As an example, lets assume we want the application traffic_server to only consume core 2 and 3, you'd simply start the application with

# taskset -c 2,3 traffic_server

Now, some applications have this built in (unfortuantely traffic_server does not, yet), and there are APIs in linux to control this (see the man page for sched_setaffinity() ). You can of course also "nice" your application accordingly, but the above lets you get a somewhat more controlled behavior (again, for threaded apps that can use more than one core).

Hacking: 

Fedora Core 15 as a VirtualBox guest OS

Fedora Core 15 was released today! I of course went straight into VirtualBox, after the download finished of course, to create a new couple of VMs. As I usually do, I assigned 512MB of RAM for the new VMs (one 32-bit and one 64-bit). Booting up, the kernel would crash, wth? It turns out, 512MB is not enough to boot up FC15, at least not under VirtualBox. I increased the memory allocated to each VM to 768MB, and things works just fine now.

Now of course to deal with the disaster that is Gnome 3.0. It's probably time to switch over to KDE, the Gnome org is just going crazy removing options and configurations for no apparent reason other than telling people they know best. I mean, come on, not even allowing me to use "Select windows when the mouse moves over them"?? Grrrr...

Update: You can achieve this particular setting ("focus follows mouse") using gconf-edit, setting

/apps/metacity/general/focus_mode

 

to "sloppy" (without the quotes). Sigh. Of course, turning on "fallback" mode is almost a requirement.

Hacking: 

Stupid benchmark

To add to the pool of braindead benchmarks, but perhaps with a little more reason, I'm adding this, and take it for what it is. If anything, this shows that performance is generally not the primary argument for choosing an intermediary. This is what I've been preaching - Yeah, performance is important, but most servers available today will handle a ridicious amount of HTTP traffic.

This is a test against an AMD Phenom(tm) II X4 940 Processor (very cheap), running across a GigE network. There are two linksys switches between the load geneating host and the server, but no routing or packet filtering. The payload is 100 bytes + a fairly small header, and the test is running with keep-alive.

In all tests, all logging is disabled.

Varnish

The configs are mostly the defaults, the main thing was I had to jack up the minimum threads, 200 seems to be a reasonable number for this test. During the test, the load goes to 300. The version of varnish is v2.1.5, from the Fedora repository.

5719306 fetches on 450 conns, 450 max parallel, 5.776500E+08 bytes, in 60 seconds
101 mean bytes/fetch
95320.7 fetches/sec, 9.627390E+06 bytes/sec
msecs/connect: 3.955 mean, 6.453 max, 0.162 min
msecs/first-response: 3.546 mean, 1005.235 max, 0.076 min

 

Nginx

This is running the older v0.8.53 version, since it's what was made available on the Fedora repo. The configs had to be tuned some, increasing the number of worker processes, setting the open_file_cache high, and also increasing the keepalive_requests setting (high).

5848823 fetches on 450 conns, 450 max parallel, 5.848820E+08 bytes, in 60 seconds
100 mean bytes/fetch
97480.4 fetches/sec, 9.748040E+06 bytes/sec
msecs/connect: 1.340 mean, 3.558 max, 0.469 min
msecs/first-response: 3.522 mean, 280.463 max, 0.067 min

 

Apache Traffic Server

This is the winner, of course, otherwise I wouldn't have published these results ;). This is running ATS v2.1.8, with mostly stock config. The primary configuration changes is to set the number of worker threads to 5 and turning off some verbose Via and server strings.

6944993 fetches on 450 conns, 450 max parallel, 6.945000E+08 bytes, in 60 seconds
100 mean bytes/fetch
115748.6 fetches/sec, 1.157486E+07 bytes/sec
msecs/connect: 1.805 mean, 2.995 max, 0.519 min
msecs/first-response: 1.736 mean, 218.573 max, 0.081 min

 

Update: I updated with the latest results from ATS v2.1.9, they are marginally different.

May 2011 Apache Traffic Server performance

I just ran a small tests against Apache Traffic Server, to see how performance has improved since last time. My test box is my desktop, an Intel Core i7-920 at 2.67Ghz (no overclocking), and the client runs on a separate box, over a cheap GigE network (two switches in between). Here are the latest results:

2,306,882 fetches on 450 conns, 450 max parallel, 2.306882E+08 bytes, in 15 seconds
100 mean bytes/fetch
153,792.0 fetches/sec, 1.537920E+07 bytes/sec
msecs/connect: 5.326 mean, 13.078 max, 0.149 min
msecs/first-response: 2.094 mean, 579.752 max, 0.099 min

 

This is of course for very small objects (100 bytes) served out of RAM cache, with HTTP keep-alive. Still respectable, close to 154k QPS out of a vey low end, commodity box.

Hacking: 

Forcing a "check" on a Linux md RAID device

To be proactive, I've found that once in a while (perhaps via a cron job) it might be a good idea to force a Linux RAID (mirror or RAID5/6) to be checked for consistency. This can easily be done from command line, with something like

% sudo echo check >  /sys/block/md0/md/sync_action

Also, to repair bad raid device, perhaps something like

% sudo echo repair >/sys/block/md0/md/sync_action

This second command solved a problem for me, where I'd get an email warnings once a week, saying "WARNING: mismatch_cnt is not 0 on /dev/md0". This was verified with

% sudo cat /sys/block/md1/md/mismatch_cnt
256

 

Hacking: 

Pages

Subscribe to RSS - leif's blog