RHEL4 and Sound Blaster Live! 24-bit

My workstation at work (yeah, I do work ...) crapped out big time on my old sound card (an old Vortex2 card) after I upgraded to RHEL4. So, I went to the local Fry's store, and picked up the cheapest Sound Blaster card I could find, the SB Live! 24-bit for a mere $29.99. Well, as it turns out, this card is not supported by the ALSA drivers provided with RHEL4...

Fortunately, it was pretty damn straight forward to upgrade the ALSA drivers on this box, these are the steps I had to take:

{CODE()}

RedHat released DS!

Finally we have an alternative to OpenLDAP: Fedora Directory Server! This is great news for everyone who's used the old Netscape or iPlaner Directory server. Not only do we get the software "for free", the source is also available!

I've just started playing with this release, and so far it's looking pretty good. I expect to have more news once I've moved more data from my old OpenLDAP directories over to FDS (Fedora Directory Server, I have no idea if that's an appropriate acronym). Head over to the Fedora Directory Server Wiki site for more information, and Downloads.

Also, I decided to "revive" an old site of mine, slapi.com, which I hope will be a dedicated resource for writing new extensions for the Directory Server. There's not much there yet, but I've re-posted an old article of mine I wrote 5-6 years ago.

Serial ATA and ABIT IS7 (ICH5-R)

Last week I built a new file server based on the ABIT IS7 mobo. I bought 2 large SATA drives, and planned to use simple mirroring of the two disks (RAID-5 just sucks too much :-). I installed Fedora Core 3 on this system, and that worked seemingly without problems. But, upon further examination, I started getting an error like this in my syslog:

Mar 31 20:33:34 server kernel: Losing too many ticks!
Mar 31 20:33:34 server kernel: TSC cannot be used as a timesource.
Mar 31 20:33:34 server kernel: Possible reasons for this are:
Mar 31 20:33:34 server kernel:   You're running with Speedstep,
Mar 31 20:33:34 server kernel:   You don't have DMA enabled for your hard disk (see hdparm),
Mar 31 20:33:34 server kernel:   Incorrect TSC synchronization on an SMP system (see dmesg).
Mar 31 20:33:34 server kernel: Falling back to a sane timesource now.

This obviously couldn't be good... Running "hdparm-t" on the disks showed me abysmal read performance from the SATA disks (like, in the 2-3MB/sec range, when I'd expect at least 60MB/sec). I spent quite a while trying to figure this, trying newer kernels (2.6.11), trying without RAID mirroring etc. I also checked the obvious BIOS settings, but the either all seemed right, or didn't make any difference. Until I tried the setting for the actual IDE controller, where the default "mode" obviously didn't do the right thing. Changing this to "Enhanced mode" did the trick, and the system performs well now.

Yahoo Search Web Services

Well, it's finally official, Yahoo released the search APIs to the public. The Developer Network has all sort of documentation, and downloads of example code and APIs. Obviously, you should use Python. Swaroop has already begun hacking, writing a QT Image search tool thingie. The Python APIs are hosted on SourceForge, at http://pysearch.sf.net/ .

I'm about 50% done with my next search project, a Gimp plugin that lets me search for Images, preview the thumbnail, and then import the selected image(s) as layers into the Gimp. I'll post with screenshots once it's looking less gimpy.

Fedora Core "devel" packages and yum

I try to keep my system up to date using the Fedora "devel" packages, using the yum updater. Sometimes I would get errors like

unsigned package //var/cache/yum/development/packages/glibc-2.3.4-3.i686.rpm
unsigned package //var/cache/yum/development/packages/eog-2.8.1-1.i386.rpm
unsigned package //var/cache/yum/development/packages/pango-1.8.0-1.i386.rpm

The problem here is that not all development packages are signed, for whatever reason. Assuming you still trust Fedora, simply modify your yum configuration file (/etc/yum.repos.d/fedora-devel.repo in my case) to add a line like
gpgcheck=0

More on snapshots

I've been using rsnapshot for quite a while now, and I'm really happy with it. It's been saving my ass on several occasions already. To make access to certain snapshots easier, I've created a small "system" of NFS mounting and symlinks.

The general idea here is to setup a number of symlinks in the directories that I've performed snapshots of. For example, in my home directory I end up having these symlinks:

.daily.3 -> /misc/.snapshots/daily.3/server3/home/leif/
.daily.2 -> /misc/.snapshots/daily.2/server3/home/leif/
.daily.1 -> /misc/.snapshots/daily.1/server3/home/leif/
.daily.0 -> /misc/.snapshots/daily.0/server3/home/leif/
.weekly.1 -> /misc/.snapshots/weekly.1/server3/home/leif/
.weekly.0 -> /misc/.snapshots/weekly.0/server3/home/leif/
.monthly.0 -> /misc/.snapshots/monthly.0/server3/home/leif/

Setting up the NFS mount point


First of all, we need to have a read-only mount point where the actual mountpoints are stored. In my case, I use an automount map for /misc, like
.snapshots      -ro                     server3:/export/.snapshots


Creating symlinks


This is all good and well, but we also need to maintain symlinks into the NFS mount point. I wrote a very simple Python script to recursively create symlinks. It's fairly configurable, using the PythonConfig style configuration file. Here's an example configuration:
[snaplinks]
base: /misc/.snapshots
exclude: daily\.[45] weekly\.2 monthly\.[1-9] quarterly\.[0-9] yearly\.[0-9]
max_depth: 1
#destination_subdir: .snapshot
 
[leif]
source: server3
directory: home/leif
destination: /home/leif
max_depth: 5
The snaplinks section provides global configurations and defaults, which you can override in individual sections. Beware, use this tool at your own risk, I've tested it quite a lot, but I'm sure there are still bugs.

Not all (Fedora) kernels are created equal

Introduction

While working on some completely unrelated piece of code, I discovered what seemed to be the impossible: FreeBSD was much faster than my 2.6 Linux system on certain system calls. And it was quite noticeable, a magnitude or more. This clearly couldn't be right, so I ended up doing some tests and I wrote a blog entry on it. Well, this continue being an issue, so I decided to expand on the topic.


Test setup

The primary test system is a Pentium 4 2.8GHz with HT, so all kernels are built with SMP support. For comparison, I've run the same tests on a few other platforms, including FreeBSD. I don't claim any of these tests to be scientifically correct in any way, but something is definitely wrong here.


Benchmarks

I checked the same code on my older system, running a 2.4 kernel on an old Celeron CPU. To my satisfaction, it was indeed very fast, even faster than FreeBSD. I double checked my PIII machine running a 2.6.6 kernel, it was also reasonably fast.

So, why was my main Linux system so damn slow? Well, as it turns out, not all kernels are created (i.e. built) equal/ I did a number of benchmarks on my system, and discovered that pretty much all my pre-built kernels (from Fedora Core 2 "rawhide") were slow, but only on a Pentium-4 system. My custom built kernel was fast

The numbers below are clock cycles per system call (smaller is better):

Kernel gettimeofday() uname() chdir() open()
FreeBSD 4.x on P4
1591
29446
8977
2583
RedHat 2.4 on Celeron
461
592
1198
864
FC2 2.6.6 on PIII
1076
1286
2547
1425
FC2 2.6.3 i686 8k stack on P4
8373
1681
33115
29725
FC2 2.6.7 i686 4k stack on P4
8453
8680
41695
37885
FC3 2.6.9 i686 on P4
8720
9058
14154
9887
custom 2.6.7 8k stack on P4
890
1031
4601
1396
custom 2.6.7 4k stack on P4
828
1002
4295
1377
custom 2.6.9 on P4
814
990
4514
1315

My custom kernel configuration is "optimized" for my particular system, the configs are available here. The simple program that I used to benchmark this is available here. This code only works on x86 platforms.


Observations


First of all, this is obviously only a problem with P4 systems, so I did a bit research (but not a lot). Apperently P4 CPUs have a new instructions for handling system calls, SYSENTER, while older Pentium systems uses the 0x80 interupt. As far as I can tell, there's a new virtual system call layer that is supposed to handle this, called vsyscall. There's some info on this in this Kerneltrap.org article. I guess I assumed that vsyscall would handle all combinations of CPUs and builds, but maybe it's not?

"Solution"

The solution for me was to recompile the "stock" Linux kernel, based on the FC3 kernel configuration file. I did quite a few changes to the config, and I also didn't apply any of the RedHat/Fedora patches to the kernel source.

The latter is the key to the solution, after trying numerous configurations and options, I narrowed it down to the set of patches Fedora applies. Dave Jones kindly confirmed this, and pointed to the exact set of patches: exec-shield. If you kernel applies the same or a similar patch, you should definitely be aware of this problem.

I haven't yet found an easy way to build the Fedora kernel without just the exec-shield patches, but I'm still working on that. Unfortunately the patches co-depend on each other a bit, so it's not as simple as just leaving out the patch.

I've filed a bug against Fedora Core for this, see Bugzilla bug #139318.


Effects

Obviously system call performance isn't a huge factor on overall system performance, although I can see certain applications suffering more than others due to this problem. To get a somewhat interesting comparison, I did a compile of the kernel source on a "slow" kernel vs my custom built one. Here are the results, output is from the time command:

"Slow" kernel:

# time gmake
real    21m10.495s
user    19m0.013s
sys     3m6.801s

"Fast" kernel:
# time gmake
real    19m17.596s
user    17m55.808s
sys     1m51.115s

Firefox and Emacs key bindings

As a die-hard Emacs users, I can't survive using my browser without Emacs bindings. After I upgraded to Firefox PR1, almost all of those bindings where MIA. The pain! Fortunately, after consulting with the good peeps at irc.mozilla.org, I was told that Firefox now honors the Gnome/GTK settings for key "themes".

Depending on your version of Gnome, you can change the setting by going to the control center and use the Keyboard Shortcuts panel. This does not seem to work in the latest Gnome version, at least my v2.8 installation of Gnome doesn't have the appropriate settings in this configuration panel.

So, in my case, I had to fire up gconf-editor manually, and edit the Key named /desktop/gnome/interface/gtk_key_theme. I changed the value from default to Emacs, and after restarting Firefox, everything was cool again.

sendmail, milters and DomainKeys

I decided to give the DomainKeys a test on my ogre.com domain the other day. My system is a pretty generic sendmail installation (8.13.1), with a few milters ("mail filters") running already. In particular, I use MIMEdefang, which likes to append a new X-Scanned-By header on all messages.

This does not work well with Domain Keys, since it expects no header changes after it calculates the signature. I could either get it to work for all my outgoing mail (the DK filter last in the milter chain), or for all incoming mail (the DK filter first). But no matter what, it can't function properly for both cases.

Instead, I ended up creating a second sendmail configuration (and process) instance. This wasn't terribly difficult, but I'll describe the steps I had to take.

1. Install sendmail 8.13 (or later), and the dk-filter milter, and follow all the instructions for setting up your ))DomainKey(( system. I added the following to my DNS server:

ogre._domainkey IN      TXT     "g=; k=rsa; t=y; p= ... <excluded, use DNS>
_domainkey      IN      TXT     "t=y; o=~; n=contact leif@ogre.com;"

2. First I fixed my existing sendmail.cf (the .mc file) by making sure it did not bind the MSA port (587), and also run the DK filter first (before any other milters):

FEATURE(no_default_msa)dnl

INPUT_MAIL_FILTER(`dk-filter', `S=unix:/var/spool/dk.socket')dnl
INPUT_MAIL_FILTER(`mimedefang', `S=unix:/var/spool/md.sock, T=S:1m;R:1m')dnl

3. Next I created a second configuration, I call it sendmail-msa.cf (and .mc), which is close to identical to my original configuration. The only changes are

define(`QUEUE_DIR', `/var/spool/mqueue-msa')dnl
DAEMON_OPTIONS(`Port=587, Name=MTA, M=Ea')dnl
FEATURE(no_default_msa)dnl

INPUT_MAIL_FILTER(`mimedefang', `S=unix:/var/spool/md.sock, T=S:1m;R:1m')dnl
INPUT_MAIL_FILTER(`dk-filter', `S=unix:/var/spool/dk.socket')dnl

You have to make sure you run the dk-filter last, so that no other milters can change or add any mail headers after the DK signature has been generated.

4. Finally, I start up a second instance of sendmail, using the new configuration:

/usr/sbin/sendmail -bd -C /etc/mail/sendmail-msa.cf -q1h

And that's it!

Another reason why Comcast sucks

I've been using Comcast cable at my house for a while, primarily because it's the only "reliable" network provider that supports my area (I live sort of in the boonies). The performance is ok downstream, but of course completely pathetic upstream.

A while back ago, we upgraded to the "professional" service, which although expensive, did give us a bit better bandwith both up- and downstream. This was a simple switch of a button somewhere in their system, no upgrades or changes on my side. Recently they introduced what they call the "Home Network", which cost less than the "professional" service, but the same bandwith.

Now, all I really care for is the bandwith, but "Home Network" in this case apparently implies that you want their firewall and wireless router. For me, this would create some headaches and lost hardware/software investments, since I've already built my house network around a Linux firewall and two existing wireless hubs (supporting 802.11g). And to make things worse, I'd have to pay a $199 "service and installation" fee, plus new hardware cost (or rental fees). All this, to get something I already have. Nice thinking, Comcast ...

I spent well over 3 hours total on the phone trying to convince them I really only wanted the higher bandwith service, and that I was willing to pay the extra monthly charge without taking any of their router/hub hardware and installation. But they completely refused, and being a stubborn ass myself, I then simply downgraded my service again to "basic" internet services.

So, instead of taking my $10 / month with no extra work on their side, they forced me to downgrade the service, and save me some more money (maybe I can get that Traxxas Revo now). I guess I could have sucked it up and take the loss on my cable modem, the $199 installation fee, and rent (or buy) the new wireless router (that I don't need), and get the extra bandwith. But it just seemed so wrong on so many levels, I just couldn't take it.

So, for now, until SBC or someone else can provide me with reasonable bandwith to my house, I'm stuck running at the lowest possible bandwith from Comcast... Kill me billy.

Sponsors

Powered by Drupal, an open source content management system