Hacking articles, tricks and tips

This is a collection if articles, links and other notes that I've collected for my personal interest. The nodes marked as "Pages" also have stories under them, this is a feeble attempt to organize these things semi-logical.

Some OSS projects I work on

Unix

Unix is the next best thing since sliced bread. I've used it for a long time now (since 1985, on a PDP-11 running BSD 2.9). I prefer Linux these days, but honestly, I really don't care that much as long as it's some reasonable flavor of Unix.

Linux

I like Linux. It's not the best thing since sliced bread, but you can't beat the price, and you get very good bang for your buck.

Not all (Fedora) kernels are created equal

Introduction

While working on some completely unrelated piece of code, I discovered what seemed to be the impossible: FreeBSD was much faster than my 2.6 Linux system on certain system calls. And it was quite noticeable, a magnitude or more. This clearly couldn't be right, so I ended up doing some tests and I wrote a blog entry on it. Well, this continue being an issue, so I decided to expand on the topic.


Test setup

The primary test system is a Pentium 4 2.8GHz with HT, so all kernels are built with SMP support. For comparison, I've run the same tests on a few other platforms, including FreeBSD. I don't claim any of these tests to be scientifically correct in any way, but something is definitely wrong here.


Benchmarks

I checked the same code on my older system, running a 2.4 kernel on an old Celeron CPU. To my satisfaction, it was indeed very fast, even faster than FreeBSD. I double checked my PIII machine running a 2.6.6 kernel, it was also reasonably fast.

So, why was my main Linux system so damn slow? Well, as it turns out, not all kernels are created (i.e. built) equal/ I did a number of benchmarks on my system, and discovered that pretty much all my pre-built kernels (from Fedora Core 2 "rawhide") were slow, but only on a Pentium-4 system. My custom built kernel was fast

The numbers below are clock cycles per system call (smaller is better):

Kernel gettimeofday() uname() chdir() open()
FreeBSD 4.x on P4
1591
29446
8977
2583
RedHat 2.4 on Celeron
461
592
1198
864
FC2 2.6.6 on PIII
1076
1286
2547
1425
FC2 2.6.3 i686 8k stack on P4
8373
1681
33115
29725
FC2 2.6.7 i686 4k stack on P4
8453
8680
41695
37885
FC3 2.6.9 i686 on P4
8720
9058
14154
9887
custom 2.6.7 8k stack on P4
890
1031
4601
1396
custom 2.6.7 4k stack on P4
828
1002
4295
1377
custom 2.6.9 on P4
814
990
4514
1315

My custom kernel configuration is "optimized" for my particular system, the configs are available here. The simple program that I used to benchmark this is available here. This code only works on x86 platforms.


Observations


First of all, this is obviously only a problem with P4 systems, so I did a bit research (but not a lot). Apperently P4 CPUs have a new instructions for handling system calls, SYSENTER, while older Pentium systems uses the 0x80 interupt. As far as I can tell, there's a new virtual system call layer that is supposed to handle this, called vsyscall. There's some info on this in this Kerneltrap.org article. I guess I assumed that vsyscall would handle all combinations of CPUs and builds, but maybe it's not?

"Solution"

The solution for me was to recompile the "stock" Linux kernel, based on the FC3 kernel configuration file. I did quite a few changes to the config, and I also didn't apply any of the RedHat/Fedora patches to the kernel source.

The latter is the key to the solution, after trying numerous configurations and options, I narrowed it down to the set of patches Fedora applies. Dave Jones kindly confirmed this, and pointed to the exact set of patches: exec-shield. If you kernel applies the same or a similar patch, you should definitely be aware of this problem.

I haven't yet found an easy way to build the Fedora kernel without just the exec-shield patches, but I'm still working on that. Unfortunately the patches co-depend on each other a bit, so it's not as simple as just leaving out the patch.

I've filed a bug against Fedora Core for this, see Bugzilla bug #139318.


Effects

Obviously system call performance isn't a huge factor on overall system performance, although I can see certain applications suffering more than others due to this problem. To get a somewhat interesting comparison, I did a compile of the kernel source on a "slow" kernel vs my custom built one. Here are the results, output is from the time command:

"Slow" kernel:

# time gmake
real    21m10.495s
user    19m0.013s
sys     3m6.801s

"Fast" kernel:
# time gmake
real    19m17.596s
user    17m55.808s
sys     1m51.115s

ABIT IC7 audio for Linux


Introduction


To begin with, it seems the native RedHat/Linux driver for the Intel 8x0 audio devices does not recognize the on-board audio chip on ABIT's motherboard. I did some research, and my web searches and documentation reading didn't give any positive results (although, others have reported that similar systems worked out of the box with the RH9 drivers). I'm guessing it's something to do with how the ABIT card is registering the audio device, like 
00:1f.5 Multimedia audio controller: Intel Corp. 82801EB AC'97 Audio (rev 02)
       Subsystem: ABIT Computer Corp.: Unknown device 1022
       Flags: bus master, medium devsel, latency 0, IRQ 17
       I/O ports at d400 [size=256]
       I/O ports at d800 [size=64]
       Memory at fc002000 (32-bit, non-prefetchable) [size=512]
       Memory at fc003000 (32-bit, non-prefetchable) [size=256]
       Capabilities: [50] Power Management version 2

It's quite possible you can get the 8x0 OSS driver to work on your system, but I wasn't in the mood to mess with that, so I gave up.

ALSA to the rescue!


After ditching the "native" 8x0 driver provided by ))RedHat((, I went to download and build ALSA (The Advanced Linux Sound Architecture) instead. However, to save some time, it's much easier to find some existing RPMs and simply install those. For instance, FreshRPMs provide an excellent library of packages built for ))RedHat(( 9. I installed the following RPMs on my system: 
kernel-smp-module-alsa-0.9.6-1.fr_2.4.20_20.9
gnome-alsamixer-0.9.3-fr1
alsaplayer-0.99.75-fr1
alsa-lib-0.9.6-1.fr
alsa-utils-0.9.6-1.fr
xmms-alsa-0.9.12-1.fr
alsa-driver-0.9.6-1.fr

Some of these packages depend on other components, but it's pretty straight forward to figure out which ones you will need. Make sure you pick the right kernel modules for ALSA, matching your existing kernel as installed on your system!

If you play MP3's while hacking along, like I do, getting the ALSA output plugin for XMMS is a must. It was straight forward to get it going, just change your XMMS to use the ALSA output plugin, and you should be ready to go.


Could this work with the ABIT IC7?


After rebooting the system, Kudzu quickly discovered my new audio device, and configured it. Yes, I finally have audio!?! Not so fast my young padawan, Gnome was still very unhappy, alsaplayer, xmms and other audio players were still all painfully silent. Have I ever mentioned I hate computers sometimes?

Not ready to give up quite yet, I fired up the alsamixer, and I noticed pretty much all of the channels were muted by default. "That can't be good!", I said to myself, so I unmuted the relevant channels, and adjusted the sound levels appropriately. Holy cow, there be sound! And the sound was good!

Once I configured the ALSA mixer properly, I made sure to save the new default configuration (to /etc, with root privs), using the alsactl command:

root # alsactl store

To assure that sound is restored properly when rebooting the system, I added a couple of lines to my /etc/modules.conf file:

alias sound-slot-0 snd-intel8x0
post-install snd-slot-0 /usr/sbin/alsactl restore >/dev/null 2>&1 || :

But what about the Gnomes?


I was still out of luck getting the Gnomie to produce any sounds at all, not even a whimper. Esound (and the esd server) was unhappy about not finding the appropriate /dev devices to play with. What's worse, there was no sound while playing my current favorite game on Linux, Neverwinter Nights. It was getting really late now, but not having sound while cleansing the plague from Neverwinter simply wasn't acceptable.

More searches on Yahoo and Google indicated that ALSA should have an OSS compatibility mode. That looked promising, so I read through more ALSA docs, and found an interesting tidbit. Adding a few more lines to modules.conf should supposedly do the trick:

alias sound-service-0-0 snd-mixer-oss
alias sound-service-0-1 snd-seq-oss
alias sound-service-0-3 snd-pcm-oss
alias sound-service-0-8 snd-seq-oss
alias sound-service-0-12 snd-pcm-oss

A quick reboot, god I love my new 3GHz P4 machine, and I have a happy Gnome again. Now for the final test, will NWN produce sound? Yes!!! I'm back in business, my Paladin can once again rid the world of all that is evil, and the peasants can rejoice.

Conclusions


I must say the Linux world still has some ways to go when it comes to audio support. It shouldn't have to be this difficult to get sound working, and I don't think I should have to pay extra money to get an audio driver that works (which I have done in the past).

ALSA seems to work extremely well so far, and I'm very pleased with it. Enabling the OSS compatibility mode was really easy, and I got the best of both worlds. Way to go ALSA people, I might even have to switch to SUSE Linux now.


Updates and user feedback


11/22/2003: XMMS and ESD


Fredrik S. Bredeli sent me an email informing me that using xmms with ESD The (Enlightened Sound Daemon) solved audio resource contention issues for him. Just make sure the ESD daemon is running properly, easiest using the Gnome configuration tools. Or, manually with something like

 esd -d /dev/dsp

Once ESD is running, just switch XMMS to use the eSound Output Plugin again.

---

-- Leif Hedstrom

Gnome configs

This is a list and descriptions of configurations I do for my Gnome setup. It's not a whole lot, I generally try to stick to the system defaults as much as possible.

Gnome general  shortcuts

These are the shortcuts I modify:

  • Switch to workspace [1-10]: Ctrl-[1-0]
  • Switch to workspace on the {left, right, above, below}: Ctrl-{Left, Right, Up, Down}

Gnome Terminal

For Gnome terminal, I modify the following shortcuts:

  • Switch to {Previous, Next} Tab: <Alt>-{Left, Right}
  • Switch to Tab [1-9]: <Alt>-[1-9]

 

I create a new profile, Small, which I use Monospace 9 font for. The Default font size is one size larger, Monospace 10. I make sure to enable "Run command as a login shell".
 

OCZ Vertex SSDs

This is a quick overview of changes I make to my Linux system using the OCZ Vertex SSD disks (which I really like).

Change I/O scheduler

In rc.local (or something similar) add a line like

echo deadline > /sys/block/sda/queue/scheduler
echo 0 > /sys/block/sda/queue/rotational

or perhaps even better

echo noop > /sys/block/sda/queue/scheduler
echo 0 > /sys/block/sda/queue/rotational

Turn off "atime" in all mounts

Edit /etc/fstab, and change the mount options (usually default) and add the noatime option. This will prevent excessive writes due to reading the disk.

Move "logs" to a tmpfs disk

To avoid lots of writes on the disks due to log activity (which I usually have a fair amount of), move /var/log to a tmpfs partition. For example

tmpfs			/var/log	  tmpfs	size=1024m	0 0

Swapiness

The default "swap" behavior will move inactive applications to swap, after some time. You can avoid this by something like

echo 0 > /proc/sys/vm/swappiness

 Making filesystems

This is untested (by me at least). First you must clear the partition table

fdisk -H 32 -S 32 /dev/sdc
mkfs.ext4 -E stripe-width=128 /dev/sdc1

Use the 'o' command in fdisk to setup a new empty partition table. Now create a partition, it's recommended to skip the first cylinder (I don't know why, but it gurantees the alignment). Remember "linux" is partition code 83.

 

Alternatively, if you want to use the entire disk (no partitions), there's no reason to try to align things. Just drop the partition table, and use mkfs.ext4 as above with the entire disk (e.g. /dev/sdc).

Creating an ext4 fs without journaling

mkfs.ext4 -O ^has_journal /dev/sdc1

or

tune2fs -O ^has_journal /dev/sdc1

 

Enabling TRIM on Linux

This probably only works on fairly modern Linux versions:

/dev/sda1 / ext4 discard,defaults

Some links

I've flashed my OCZ drives pretty easily to v1.5. Make sure you follow the directions properly. Since I'm on Linux, I simply made a FreeDOS boot USB, and copied over all the flash .exe's to it. I have to run the SSDCHK.EXE manually (since I'm not using the autoexec.bat etc.).

Wiper.sh

This dangerous script is available with later versions of hdparm. With the appropriate firmware support (v1.5 supports it it seems), you can TRIM unused blocks in the SSD, to restore performance. The automatic garbage collection ought to do this for you, but I have no idea if/when you actually need to run wiper.sh manually. Definitely use on your own risk (I've only tried it once).

Tweaking XFCE

This is a collection of things I've tweaked during my transition from Gnome (because Gnome 3.0 is useless) to XFCE. This is still work in progress, but comments and feedback is much appreciated.

Terminal

The default terminal with XFCE has some weird behavior, and even though most of it can be tweaked in the GUI prefs, some can not. In particular, I made the following changes to ~/.config/Terminal/terminalrc:

MiscCursorBlinks=TRUE
MiscCursorShape=TERMINAL_CURSOR_SHAPE_BLOCK
ColorCursor=#0000aaaa0000

This allows me to better see the letter that the cursor is currently over. Note that I do green text on black background on all my Terminals, it's a habit from the VT100 days at the University in '85...

VirtualBox


News

  • VirtualBox 2.10 is released, fresh Downloads available now for many distros and OS's.

Host tips

A few useful commands / tips for managing the Host side of VirtualBox.


Shrinking a guest image (VDI)


Simply run the command

$ VBoxManage modifyvdi  some_image.vdi compact

However, this might not accomplish much, if the guest OS has "garbage" written all over the unused sectors. You can "fix" this by zeroing out all free disk space on all partitions, for example do something like inside the guest (assuming it's a linux OS):

$ sudo cat /dev/zero> /fill.it
$ sync
$ sudo rm -f /fill.it
$ sync
$ sudo halt

On a VDI I had which used 6.4GB on the host, and only 3GB inside the guest, doing the above shrunk the VDI down to 4.4GB. Not great, but a lot better


Guest OS tips


Here's a small section of some tips to get VirtualBox to work well with certain guest OS's.

Debian


In order to build the guest additions, you must install some required packages. I believe something like this would do it (run it as root):

$ apt-get install build-essential module-assistant
$ m-a prepare

This might also work / help:

$ apt-get install build-essential linux-headers-`uname -r`

Taking snapshots

I have a fairly large amount of data (source code, DB dumps, docs etc.) that I keep on either my workstations or a file servers. I use software RAID on both systems, either mirroring (RAID1) or stripes+parity (RAID5), and that obviously saves me from fatal disk errors. But this doesn't prevent me from losing data when I'm a total moron, or some application goes bad.

So a while ago, using some favorite tools (Yahoo search obviously, for those of you who know me) I went out to see what was out there. I found this very informative article on the topic. Doing some more searches, I then found a little nugget called rsnapshot. This tool pretty much automates everything necessary to perform hourly,daily, weekly, monthly or whatever type of snapshots you wish to do.

Configuring rsnapshot is pretty straight forward, and it comes with a good template configuration file that you can tune and tweak. It's assumed you have a recent version of rsync installed, and SSH properly setup and running if you are doing snapshots over the wire. I'll describe a few of the configurations that I've used. First off, you need to provide some basic information about how snapshot should behave, where to store snapshots etc. Note: this is for a Linux system:

snapshot_root   /export/.snapshots/
cmd_cp          /bin/cp
cmd_rsync       /usr/bin/rsync
cmd_ssh         /usr/bin/ssh
link_dest       1
verbose         3
loglevel        3

Next we need to decide what types of snapshots we want, and what sort of retention to keep. I've decided to do daily, weekly and monthly snapshots only, keeping 6 daily, 3 weekl, 3 monthly snaps, 3 quarterly and 9 yearly (yeah ...).

interval        daily   6
interval        weekly  3
interval        monthly 3
interval        quarterly 3
interval        yearly 9

Finally, we need to specify which directories to make snapshots of, possibly from a remote server. In my case, I do snapshots over the network only, to keep all snapshots on a RAID5 device.

# Workstation
backup  root@ws1.ogre.com:/etc/        ws1/etc/       exclude_file=/admin/etc/ws1.exclude
backup  root@ws1.ogre.com:/export/     ws1/export/    exclude_file=/admin/etc/ws1.exclude
 
# Web/mail server
backup  root@s1.ogre.com:/etc/         s1/etc/        exclude_file=/admin/etc/s1.exclude
backup  root@s1.ogre.com:/data/        s1/server/     exclude_file=/admin/etc/s1.exclude

This part of the configuration is a bit finicky, in particular, you have to use a &lt;TAB&gt; character between the destination directory (e.g. ws1/etc/) and any extra options you want to pass to rsync. Any &lt;SPACE&gt; characters will actually be part of the directory name, which was kind of a surprise to me.

With this all configured, you're pretty much set to go, just run rsnapshot out of your crontab at the desired frequency. In my case, since I do daily snapshots (and not hourly), I just added a daily cron job, like:

#!/bin/sh
                                                                                        
rsnap='/usr/local/bin/rsnapshot'
                                                                                        
do_weekly=0
do_monthly=0
do_quarterly=0
do_yearly=0
                                                                                        
wday_num=`/bin/date '+%u'`
if [ $wday_num -eq 7 ]; then
    do_weekly=1
    day_num=`/bin/date '+%d'`
    if [ $day_num -ge 25 ]; then
        do_monthly=1
        m_num=`/bin/date '+%m'`
        if [ $m_num -eq 3 -o $m_num -eq 6 -o $m_num -eq 9 -o $m_num -eq 12 ]; then
            do_quarterly=1
        fi
        if [ $m_num -eq 12 ]; then
            do_yearly=1
        fi
    fi
fi
                                                                                        
if [ $do_yearly -eq 1 ]; then
    echo "Saving yearly snapshot"
    $rsnap yearly
fi
if [ $do_quarterly -eq 1 ]; then
    echo "Saving quarterly snapshot"
    $rsnap quarterly
fi
if [ $do_monthly -eq 1 ]; then
    echo "Saving monthly snapshot"
    $rsnap monthly
fi
if [ $do_weekly -eq 1 ]; then
    echo "Saving weekly snapshot"
    $rsnap weekly
fi
                                                                                        
echo "Doing daily snapshot"
$rsnap daily

This will run daily snapshots Monday - Saturday, a weekly on Sundays, except on the last Sunday of the month I perform a monthly snapshot. I also NFS export my snapshot directory (/export/.snapshots), read-only, so that I can easily get to it from all my machines.

PulseAudio

Pulse Audio is the new audio server software shipped with several modern Linux distros (Ubuntu, Fedora Core etc.). This is a little page I'm writing, to collect useful information about this system. Pulse Audio replaces older audio server systems like esound (gnome) and aRts (KDE), at least I hope they will.

Features



Installation and usage



Volume control



Other tips



Running non-pulseaudio applications


Since PulseAudio grabs the sound device, you can not have non-PA enabled applications use the sound driver. This can be worked around by simply suspending PulseAudio for a moment, using the pasuspender application. For example, to run audacity, I do
$ pasuspender -- audacity

Disk Recovery

Dell Mini-9 netbook

This is my latest toy, and I figured I need a place to collect useful tips and links. I really like this thing, although, my unit might have a heat issue. I'll know more as I progress, but definitely keep an eye on your Dell mini-9. If the performance benchmarks aren't consistent, you should probably get it replaced.

Useful links

  • The MyDellMini sites have lots of useful information about the hardware, OSes etc. 
  • To test the performance, I installed and ran GeekBench. This little gem runs on Linux, Windows and MacOS X!
  • Official Dell site for the mini.

 

Ubuntu configuration changes

First off, and to my surprise, the SSD disk is mounted with default mount options. This, I believe, implies updating access timestamp. To avoid this, I changed the /etc/fstab entry, to include a -noatime option for the relevant disk partitions / mount points.

Secondly, it's my belief that SSD drives fare better with the NoOp scheduler. I added the following to my startup script:

echo noop > /sys/block/sda/queue/scheduler

Finally, I've had to force some WiFi setting, in order for SSH (OpenSSH server and client) to work properly. I added the following to /etc/network/if-pre-up.d/wireless-tools

/sbin/iwpriv eth1 set_vlanmode 0

 

SSD cards

I (unfortunately) got the Super Talent 32GB SSD card, and although it works just fine, I wish I hadn't bought this. Instead, spend the extra $40 and get a RunCore 32GB card instead.

HTTP

I work with HTTP, a lot. Here's a few good links to useful information related to HTTP and TCP, primarily RFCs etc.

These are not directly related to HTTP, but nevertheless useful:

 

MacOSX

Python

Python is seriously one of the best scripting language today.

Python performance through times

I recently compiled all Python version from v2.2 to 3.0b, to see how their performance compares. I decided to not use pybench, but to take some of the benchmarks from the Computer Language Benchmarks Game instead (hoping they are slightly more "real use" realistic). I compiled all versions of Python identically, using the same compiler (4.3.0) and the same optimization options ("-O3 -march=core2 -mtune=core2"). All benchmarks were run 20 times for every python version, and the fastest run for each benchmarks and interpreter was picked. This obviously gives a "best case" scenario (I think), the other alternative would be to do a median or average, but I wanted to avoid any unfairness due to system/OS activities.

The benchmarks had to be ported to support Python3000 (v3.0b3), but these changes were mostly trivial (print's and xrange's), so I don't think that should affect the results. My test system (a Core2 Duo box with plenty of RAM) was "unused" during the entire test run (which took over 6 hours to complete). Alright, so what are the results? The most interesting data is the relative performance index. This is the average of each test as compared to Python v2.2.3, which therefore has an index of "1.0". This also means that each test has equivalent weight in the total index calculation (a higher index is better).


py-performance-index.png



I'm also including the results for each individual benchmark, in the following graph (times in seconds, lower is better):


py-performance-bench.png

Update: On request from a friend, I tried compiling with "-Os" instead of "-O3", and not surprisingly, compiling for size is not advantageous on my Core2 box. This is in line with the results from the Firefox tests I did before. Again, the 4MB L2 cache probably negates any benefits from compiling for size.

I'm not going to make any comments about what might have happened after v2.4.x, but it's good to see that Python3k is getting very promising results.

Web stuff

Ogre sites

Site management

Site admin tools

Here's a short list of useful Webmaster sites:

Mail

I work (and have worked) on many E-mail related projects, including the anti-spam systems used at Yahoo. I've written a few pages on various email topics here.

Checking your IP or domain's status

This is a small collection of URLs and sites you can use to see if your IP or domain is considered "spammish" or not. This is not a complete list, but I'll update it as I find more useful tools.

Senderbase


This is an IP clearinghouse run by IronPort, and used by many commercial anti-spam systems (like, Barracuda). You can check your IP or domain status here.

RBL checks


There are plenty of RBL systems out there, and plenty of tools to check your status. Here are a few:

Anti-Spam and email tools

This is a short list of tools I use to manage my mail servers (and mailboxes). The main building blocks for my system are

sendmail


I'm running sendmail v8.13.1 currently, supporting a number of domains and users. sendmail uses the Cyrus mailer for delivery, which in turns talks to an LMTP server. This injects the new messages directly into the IMAP folders. As an extra benefit, I also get the + style addressing support automatically, meaning I can sort into sub-folders by the envelope address. For instance, if you send an email to 
leif+other.spam@ogre.com

it is automatically sorted into my IMAP folder named other.spam.


Cyrus IMAP


Cyrus has been my choise of IMAP server for quite a while, way back when there was really no comparison in features and performance. Netscape Mail server was very promising, but unfortunately died during the AOL and Sun massacres. Cyrus supports SASL of course, but also a somewhat unknown sub-system called Sieve. This is a 100% server side mail filtering language, and it's quite powerful, much more so than I can document here. But, here's an example filter file:

require "fileinto";
 
if header :is "X-Spam-Flag" ["YES"] {
    fileinto "INBOX.zSpam";
} elsif address :is :all ["from", "to", "cc", "bcc"] ["foo@ogre.com", "bar@ogre.com"] {
    fileinto "INBOX.filtered.giants";
} elsif address :is :all ["to", "cc", "bcc"] ["dist-update@perldap.org"] {
    fileinto "INBOX.filtered.dist";
} elsif header :contains "List-Id" "some_mail_list" {
    fileinto "INBOX.mailists.games";
}


Spam tools


I use a few sendmail milters to help get rid some of the worst spam:

I'm fully aware that MIMEdefang supports SpamAsassasin, but I prefer to run these two systems independently. In fact, I wish MIMEdefang would make it easier to disable the user of SA! For SpamAssassin I use pretty much every bell and whistle there is, including:

  • DCC
  • Pyzor
  • Razor


Anti-virus


Although MIMEDefang does some AV already, it also supports "pluggable" virus detectors. I currently use

sendmail, SASL, TLS and SSL


Project goal


When I started this project, my primary goal was to allow for authenticated SMTP over a secure channel. I use sendmail as my MTA, not because I'm particularly fond of it, but because it's what I know, and it works well for me. My setup is pretty straight forward:
  • ))RedHat(( linux (mix of 8.0 and 9.0)
  • sendmail 8.12.10
  • Latest OpenSSL and SASL libraries (I gave up on using RHs outdated builds)
  • A myriad of milters, spam filters, and tools around sendmail and Cyrus IMAP.

I'm not going to go into details on the exact mail server configurations, but I'm concentrating on the changes I made to enable SSL and SASL for sendmail.


Clients and TLS vs SSL-only


We mostly use well behaved mail clients for MUAs, but my wife insists on using Entourage on her Mac. To her defense, there's not a lot of alternatives out there for Macs, Mozilla isn't working very well on her system. Being a Microsoft client, Entourage of course doesn't follow the specifications. In particular, it doesn't support the STARTTLS command. The client would simply complain that the server didn't provided any SASL authentication method that it could use:

The SMTP server for "Ogre" does not recognize any of the authentication
methods supported by Entourage. To send mail, try disabling SMTP
authentication in the account settings or talk to your administrator.

An unknown error (5530) occurred

The lack of proper SASL support in Entourage forced me to also include support for a dedicated SSL/TLS SMTP server (SMTPS). Fortunately, sendmail allows me to serve both the regular SMTP port (25) and SMTPS (port 465) using the same configuration.

As if this wasn't enough, once I got past this problem, Entourage misinterpreted the sendmail greetings as if client authentication (certificate) was required. I would get a client error like:

Security failure. Personal certificate required.

Fortunately sendmail provides a configuration solution for this as well, naturally. But seriously, why does MS have to make my life miserable all the time? I don't even hate them, I just prefer not to use most of their junk...


Rebuilding sendmail with SASL support


This was the easy part, simply modify your devtools/Site/site.config.m4 file to something like:

APPENDDEF(`confENVDEF', `-DSASL=2 -DSTARTTLS')
APPENDDEF(`conf_sendmail_LIBS', `-lsasl2 -lssl -lcrypto')
APPENDDEF(`confLIBDIRS', `-L/usr/local/lib -R/usr/local/lib')
APPENDDEF(`confINCDIRS', `-I/usr/local/include')

APPENDDEF(`conf_sendmail_ENVDEF', `-D_FFR_SMTP_SSL')

You'll obviously have to change your directories etc. to match your build directory. The last configuration option enables an (experimental?) support for the SSL-only daemon support. I'm not 100% sure, but I think support for SMTPS is enabled by default in sendmail 8.13. In any case, this feature is important to work around the lack of proper TLS support in Entourage (which our new sendmail daemon will announce).

Make sure to save a copy of your old sendmail binary before installing this new build. Nothing worse that not being able to do a quick rollback on your oh so important mail servers.


SSL certificates


I'm not going to go into detail on how to generate the appropriate SSL certificates here, there are plenty of sites that describes that. For example:

I personally just use my own root-CA, so I can avoid the Verisign tax. And this is plenty secure for all my needs.


Reconfigure sendmail


First of all, we need to configure sendmail to find all the required certificate information. Assuming you are using the M4 macro packages for configuring sendmail, add something like this to your sendmail.mc:

define(`confCACERT_PATH',`/usr/local/SSL/private')dnl
define(`confCACERT',`/usr/local/SSL/private/CAcert.pem')dnl
define(`confSERVER_CERT',`/usr/local/SSL/certs/sendmail-cert.pem')dnl
define(`confSERVER_KEY',`/usr/local/SSL/certs/sendmail-key.pem')dnl

Obviously, you must change the paths here. Next, we need to configure SASL properly, and here's the first stumbling block trying to support Entourage. As far as I can tell, you must have support for the LOGIN authentication mechanism enabled. This might require a recompile of your SASL libraries, since it's an optional feature. My sendmail.mc adds the following lines for SASL:

define(`confAUTH_MECHANISMS',`LOGIN PLAIN')dnl
TRUST_AUTH_MECH(`LOGIN PLAIN)dnl
define(`confDONT_BLAME_SENDMAIL', `GroupReadableSASLDBFile')dnl

I only support "plain" SASL authentication, but other authentication mechanisms includes CRAM-MD5 and DIGEST-MD5. Using such mechanism requires you to setup and manage the SASL authentication database properly. The last configuration eliminates a sendmail runtime warning, if your SASL DB is group readable (common, to give sendmail read permissions?).

Entourage still won't talk to this sendmail configuration, because of it's broken STARTTLS implementation. The only workaround I know of is to enable the SSL-only port/daemon. This is done with a few more configuration lines to your sendmail.mc:

DAEMON_OPTIONS(`Port=smtp, Name=MTA')dnl
DAEMON_OPTIONS(`Port=465, Name=SSA, M=Eas')dnl
define(`confAUTH_OPTIONS', `A,p,y')dnl

The last option isn't absolutely necessary, but it prevents people from accidentally trying to use SMTP Authentication over a non-secure channel. It also denies anonymous logins. Alright, we're almost there. Now there's only one last problem, Entourage still complains about the requirement for certificate based authentication. Once again, sendmail provides configurations options for this:

define(`confTLS_SRV_OPTIONS',`V')dnl


Conclusions


If it hadn't been for Entourage (and I'm guessing, Outlook), enabling SASL and TLS for sendmail is pretty straight forward. I had it up and running for most of my clients in less than an hour, but figuring out all the tweaks for the rouge MUAs added several more hours of debugging and reconfigurations.

Is it worthwhile doing this? Absolutely! With SASL enabled (over TLS hopefully), your users can send email through your system from anywhere. One major benefit is that an authentication SMTP sessions is allowed to "gateway" mail just as if the client came from one of your blessed (internal) networks. And spammers can not abuse is, unless they can compromise your authentication mechanisms (i.e. hacking accounts).


Credits


Many thanks to my wife Michelle for using Entourage, and forcing me into fixing all this crud ... :-)

-- Leif Hedstrom

Mozilla

Mozilla provides a couple of tools that I use on a regular basis.

Useful extensions

Mozilla and related tools


First off, you need to get the latest Mozilla build. As an alternative, there are two standalone applications, Mozilla Firebird for browsing and Mozilla Thunderbird for reading mail.

Major apps


Firefox plugins and extensions


Greasemonkey scripts


  • ...

Thunderbird extensions


Firefox performance vs compiler options

I've compiled the latest Firefox source (nightly) with various gcc compiler optimization options, compared to the "default" moz build options on my platform. This is not a comprehensive test, but merely gives some ideas of where the "best bang for the buck" is. I ran the Sunspider benchmarks three times for each build instance. My box is a single CPU core2 system with 4GB RAM. The compiler options used were as follow: 

-Os -march=core2 -mtune=core2
-O2 -march=core2 -mtune=core2
-O3 -march=core2 -mtune=core2
-O2

The last is the default optimization options on my platform, which is what I call the "Moz" build. The "core2" options turns out to have little effect (1.01x - 1.03x at the most), but "-O3" vs "-O2" has significant impact ("-O3" always generates the fastest browser on my platform). And of course, the new JIT compiler is very fast. For now, all I have is this little screen shot of a table I made with the results, I'll convert this to a proper HTML table in my copious spare time.

The way to read this table is a little bit complicated, but pick a row (e.g. "O3 + JIT") and follow it to the column you wish to compare it to (e.g. Moz + JIT), and you see that the "O3 + JIT" build is 1.10x faster. The conclusions I can make is that it's always best to optimize as much as possible, optimizing for size doesn't makes sense on these modern CPUs (core2), for the Firefox code base. And JIT is always vastly superior.

As a side note, the build with the "-Os" options (optimize for size) was incredibly unstable with the JIT compiler enabled. It took me many, many runs to get it to complete three benchmarks without hanging.

Mobile and Portable devices

I really like my iPhone, and it has replaced my PSP as a portable media device.

Kindle

Places to Get Ebooks

Amazon specific search - This search removes all the public domain books and sorts price from low to high.
Amazon Special Offers - All the Kindle special offers on Amazon
Manybooks - Kindle format for downloads. Foreign language books as well. Has public domain books too.
Manybooks (Kindle edition) - Access through the Kindle web browser to download directly to Kindle. Choose Mobipocket format.
Feedbooks - Kindle format for downloads. Has public domain books too and I think others.
Project Gutenberg - Multiple languages. Seems to be in HTML or Text format (download text for the Kindle)
Free Kindle Books - a bunch of Project Gutenberg books in Kindle format.
Fictionwise - .mobi format. Some books free, some not, but some sales as well on the non-free ones.
World Public Library - all PDFs. Cost is $8.95/year. Over 400,000 classic titles.
Fictionpress - mostly original works, in Text format.

Conversion Software

Calibre - converts to/from a bunch of different formats including PDF and Kindle. Works on Windows, Mac, Linux.
MobiPocket - converts from PDF/Word/Text to .prc which can be read by the Kindle. Works on Windows.
Stanza - export stuff to Kindle format. Probably others, but not sure which. Works on Windows, Mac.

Ripping for PSP/iPhone from DVD

These tips make great PSP movies from DVDs

Install and Rip your DVD using dvddecrypter


  1. Use IFO mode
  2. Set it to select main movie only, and File Splitting to "none".
  3. Set it to make 1 VOB.
  4. Set it to ignore hardware read errors.

Install AutoGK. Some options you'll have to go to the advanced tab for


  1. use AutoGK to create an AVI from the VOB.
  2. set width to 400
  3. Use Xvid
  4. set quality to 75 to 80 (don't go over 80, it doesn't give you much).
  5. There are not alot of options.

Convert to MP4 from AVI using PSP Video 9


  1. Use the 400x192 resolution
  2. Use the following ffmpeg options -g 300 -sameq
  3. set for 2 pass
  4. set audio to 80k to 96k (IMO), Stereo.
  5. I use a Framerate of 14.985 FPS, and a bitrate of 512kbps (400kbps is also decent).
  6. If you have a PSP with 2.50 (or later) firmware, maybe consider the AVC video codec, but it takes a lot more space.
  7. Tweaking the sound volume over 100% might help on the PSP, but I've had little luck with it. I bought a Bostaroo instead, works great.480

Conferences

Here's a list of interesting and upcoming conferences.

Network


Small collection of some useful network and TCP tidbits.

TCP socket options

Some collected useful tips about various TCP socket options, e.g. setsockotp() and getsockopt().

TCP_DEFER_ACCEPT

This is availble on Linux, and is a poor mans "accept filtering" (as FreeBSD implements it). Apache HTTPD v2.2.x sets this to "1" (which I assume was a mistake), which translates to 3s before timing out. HTTPD 2.3.x increases this to 30s, which according to the table I found for Linux v2.6.32 (and later) translates to 45s. Internally, the kernel translates the timeout to a number of retransmits to perform before the timeout, which means a call to setsockopt() followed by getsockopt() won't necessarily show the same value. I found the following conversions, from the requested timeout (in seconds) to effective timeout as used internally):

1-3	-> 3
4-9	-> 9
10-21	-> 21
22-45	-> 45
46-93	-> 93
94-189	-> 189
190-309	-> 309
310-429	-> 429

On older kernels, I see the following conversion (due to different rounding I think):

1-3	-> 3
4-6	-> 6
7-12	-> 12
13-24	-> 24
25-48	-> 48
49-96	-> 96
97-192	-> 192
193-384	-> 384

The code for this is available here.