Hacking articles, tricks and tips

This is a collection if articles, links and other notes that I've collected for my personal interest. The nodes marked as "Pages" also have stories under them, this is a feeble attempt to organize these things semi-logical.

Some OSS projects I work on

  • Apache Traffic Server - A fast, complete proxy and caching server.
  • pYsearch - Yahoo search APIs for Python (pretty much killed off by Y!  :/).



Unix is the next best thing since sliced bread. I've used it for a long time now (since 1985, on a PDP-11 running BSD 2.9). I prefer Linux these days, but honestly, I really don't care that much as long as it's some reasonable flavor of Unix.



I like Linux. It's not the best thing since sliced bread, but you can't beat the price, and you get very good bang for your buck.

Not all (Fedora) kernels are created equal


Gnome configs

This is a list and descriptions of configurations I do for my Gnome setup. It's not a whole lot, I generally try to stick to the system defaults as much as possible.

Gnome general  shortcuts

These are the shortcuts I modify:

  • Switch to workspace [1-10]: Ctrl-[1-0]
  • Switch to workspace on the {left, right, above, below}: Ctrl-{Left, Right, Up, Down}

Gnome Terminal

For Gnome terminal, I modify the following shortcuts:

  • Switch to {Previous, Next} Tab: <Alt>-{Left, Right}
  • Switch to Tab [1-9]: <Alt>-[1-9]


I create a new profile, Small, which I use Monospace 9 font for. The Default font size is one size larger, Monospace 10. I make sure to enable "Run command as a login shell".

OCZ Vertex SSDs

This is a quick overview of changes I make to my Linux system using the OCZ Vertex SSD disks (which I really like).

Change I/O scheduler

In rc.local (or something similar) add a line like

echo deadline > /sys/block/sda/queue/scheduler
echo 0 > /sys/block/sda/queue/rotational

or perhaps even better

echo noop > /sys/block/sda/queue/scheduler
echo 0 > /sys/block/sda/queue/rotational

Turn off "atime" in all mounts

Edit /etc/fstab, and change the mount options (usually default) and add the noatime option. This will prevent excessive writes due to reading the disk.

Move "logs" to a tmpfs disk

To avoid lots of writes on the disks due to log activity (which I usually have a fair amount of), move /var/log to a tmpfs partition. For example

tmpfs			/var/log	  tmpfs	size=1024m	0 0


The default "swap" behavior will move inactive applications to swap, after some time. You can avoid this by something like

echo 0 > /proc/sys/vm/swappiness

 Making filesystems

This is untested (by me at least). First you must clear the partition table

fdisk -H 32 -S 32 /dev/sdc
mkfs.ext4 -E stripe-width=128 /dev/sdc1

Use the 'o' command in fdisk to setup a new empty partition table. Now create a partition, it's recommended to skip the first cylinder (I don't know why, but it gurantees the alignment). Remember "linux" is partition code 83.


Alternatively, if you want to use the entire disk (no partitions), there's no reason to try to align things. Just drop the partition table, and use mkfs.ext4 as above with the entire disk (e.g. /dev/sdc).

Creating an ext4 fs without journaling

mkfs.ext4 -O ^has_journal /dev/sdc1


tune2fs -O ^has_journal /dev/sdc1


Enabling TRIM on Linux

This probably only works on fairly modern Linux versions:

/dev/sda1 / ext4 discard,defaults

Some links

I've flashed my OCZ drives pretty easily to v1.5. Make sure you follow the directions properly. Since I'm on Linux, I simply made a FreeDOS boot USB, and copied over all the flash .exe's to it. I have to run the SSDCHK.EXE manually (since I'm not using the autoexec.bat etc.).


This dangerous script is available with later versions of hdparm. With the appropriate firmware support (v1.5 supports it it seems), you can TRIM unused blocks in the SSD, to restore performance. The automatic garbage collection ought to do this for you, but I have no idea if/when you actually need to run wiper.sh manually. Definitely use on your own risk (I've only tried it once).

Tweaking XFCE

This is a collection of things I've tweaked during my transition from Gnome (because Gnome 3.0 is useless) to XFCE. This is still work in progress, but comments and feedback is much appreciated.


The default terminal with XFCE has some weird behavior, and even though most of it can be tweaked in the GUI prefs, some can not. In particular, I made the following changes to ~/.config/Terminal/terminalrc:


This allows me to better see the letter that the cursor is currently over. Note that I do green text on black background on all my Terminals, it's a habit from the VT100 days at the University in '85...

Taking snapshots

I have a fairly large amount of data (source code, DB dumps, docs etc.) that I keep on either my workstations or a file servers. I use software RAID on both systems, either mirroring (RAID1) or stripes+parity (RAID5), and that obviously saves me from fatal disk errors. But this doesn't prevent me from losing data when I'm a total moron, or some application goes bad.

So a while ago, using some favorite tools ([http://search.yahoo.com/|Yahoo search] obviously, for those of you who know me) I went out to see what was out there. I found this very [http://www.mikerubel.org/computers/rsync_snapshots/|informative article] on the topic. Doing some more searches, I then found a little nugget called [http://www.rsnapshot.org/|rsnapshot]. This tool pretty much automates everything necessary to perform hourly,daily, weekly, monthly or whatever type of snapshots you wish to do.

Configuring rsnapshot is pretty straight forward, and it comes with a good template configuration file that you can tune and tweak. It's assumed you have a recent version of rsync installed, and SSH properly setup and running if you are doing snapshots over the wire. I'll describe a few of the configurations that I've used. First off, you need to provide some basic information about how snapshot should behave, where to store snapshots etc. __Note__: this is for a Linux system:

snapshot_root /export/.snapshots/
cmd_cp /bin/cp
cmd_rsync /usr/bin/rsync
cmd_ssh /usr/bin/ssh
link_dest 1
verbose 3
loglevel 3

Next we need to decide what types of snapshots we want, and what sort of retention to keep. I've decided to do daily, weekly and monthly snapshots only, keeping 6 daily, 3 weekl, 3 monthly snaps, 3 quarterly and 9 yearly (yeah ...).

interval daily 6
interval weekly 3
interval monthly 3
interval quarterly 3
interval yearly 9

Finally, we need to specify which directories to make snapshots of, possibly from a remote server. In my case, I do snapshots over the network only, to keep all snapshots on a RAID5 device.

# Workstation
backup root@ws1.ogre.com:/etc/ ws1/etc/ exclude_file=/admin/etc/ws1.exclude
backup root@ws1.ogre.com:/export/ ws1/export/ exclude_file=/admin/etc/ws1.exclude

# Web/mail server
backup root@s1.ogre.com:/etc/ s1/etc/ exclude_file=/admin/etc/s1.exclude
backup root@s1.ogre.com:/data/ s1/server/ exclude_file=/admin/etc/s1.exclude

This part of the configuration is a bit finicky, in particular, you have to use a <TAB> character between the destination directory (e.g. ws1/etc/) and any extra options you want to pass to rsync. Any <SPACE> characters will actually be part of the directory name, which was kind of a surprise to me.

With this all configured, you're pretty much set to go, just run rsnapshot out of your crontab at the desired frequency. In my case, since I do daily snapshots (and not hourly), I just added a daily cron job, like:




wday_num=`/bin/date '+%u'`
if [ $wday_num -eq 7 ]; then
day_num=`/bin/date '+%d'`
if [ $day_num -ge 25 ]; then
m_num=`/bin/date '+%m'`
if [ $m_num -eq 3 -o $m_num -eq 6 -o $m_num -eq 9 -o $m_num -eq 12 ]; then
if [ $m_num -eq 12 ]; then

if [ $do_yearly -eq 1 ]; then
echo "Saving yearly snapshot"
$rsnap yearly
if [ $do_quarterly -eq 1 ]; then
echo "Saving quarterly snapshot"
$rsnap quarterly
if [ $do_monthly -eq 1 ]; then
echo "Saving monthly snapshot"
$rsnap monthly
if [ $do_weekly -eq 1 ]; then
echo "Saving weekly snapshot"
$rsnap weekly

echo "Doing daily snapshot"
$rsnap daily

This will run daily snapshots Monday - Saturday, a weekly on Sundays, except on the last Sunday of the month I perform a monthly snapshot. I also NFS export my snapshot directory (/export/.snapshots), __read-only__, so that I can easily get to it from all my machines.


Disk Recovery


Dell Mini-9 netbook

This is my latest toy, and I figured I need a place to collect useful tips and links. I really like this thing, although, my unit might have a heat issue. I'll know more as I progress, but definitely keep an eye on your Dell mini-9. If the performance benchmarks aren't consistent, you should probably get it replaced.

Useful links

  • The MyDellMini sites have lots of useful information about the hardware, OSes etc. 
  • To test the performance, I installed and ran GeekBench. This little gem runs on Linux, Windows and MacOS X!
  • Official Dell site for the mini.


Ubuntu configuration changes

First off, and to my surprise, the SSD disk is mounted with default mount options. This, I believe, implies updating access timestamp. To avoid this, I changed the /etc/fstab entry, to include a -noatime option for the relevant disk partitions / mount points.

Secondly, it's my belief that SSD drives fare better with the NoOp scheduler. I added the following to my startup script:

echo noop > /sys/block/sda/queue/scheduler 

Finally, I've had to force some WiFi setting, in order for SSH (OpenSSH server and client) to work properly. I added the following to /etc/network/if-pre-up.d/wireless-tools

/sbin/iwpriv eth1 set_vlanmode 0


SSD cards

I (unfortunately) got the Super Talent 32GB SSD card, and although it works just fine, I wish I hadn't bought this. Instead, spend the extra $40 and get a RunCore 32GB card instead.


I work with HTTP, a lot. Here's a few good links to useful information related to HTTP and TCP, primarily RFCs etc.

These are not directly related to HTTP, but nevertheless useful:



Some OS X command line utilities

I found a couple of useful page showing some handy OSX command line tricks;



Python is seriously one of the best scripting language today.

Python performance through times

I recently compiled all Python version from v2.2 to 3.0b, to see how their performance compares. I decided to not use pybench, but to take some of the benchmarks from the [http://shootout.alioth.debian.org/gp4/|Computer Language Benchmarks Game] instead (hoping they are slightly more "real use" realistic). I compiled all versions of Python identically, using the same compiler (4.3.0) and the same optimization options ("-O3 -march=core2 -mtune=core2"). All benchmarks were run 20 times for every python version, and the fastest run for each benchmarks and interpreter was picked. This obviously gives a "best case" scenario (I think), the other alternative would be to do a median or average, but I wanted to avoid any unfairness due to system/OS activities.

The benchmarks had to be ported to support Python3000 (v3.0b3), but these changes were mostly trivial (__print__'s and __xrange__'s), so I don't think that should affect the results. My test system (a Core2 Duo box with plenty of RAM) was "unused" during the entire test run (which took over 6 hours to complete). Alright, so what are the results? The most interesting data is the relative performance index. This is the average of each test as compared to Python v2.2.3, which therefore has an index of "1.0". This also means that each test has equivalent weight in the total index calculation (a higher index is better).

::{img src=http://www.ogre.com/files/ogre.com/py-performance-index.png}::

I'm also including the results for each individual benchmark, in the following graph (times in seconds, lower is better):

::{img src=http://www.ogre.com/files/ogre.com/py-performance-bench.png}::

__Update__: On request from a friend, I tried compiling with "-Os" instead of "-O3", and not surprisingly, compiling for size is not advantageous on my Core2 box. This is in line with the results from the Firefox tests I did before. Again, the 4MB L2 cache probably negates any benefits from compiling for size.

I'm not going to make any comments about what might have happened after v2.4.x, but it's good to see that Python3k is getting very promising results.


Web stuff

Site management


I work (and have worked) on many E-mail related projects, including the anti-spam systems used at Yahoo. I've written a few pages on various email topics here.

Checking your IP or domain's status

This is a small collection of URLs and sites you can use to see if your IP or domain is considered "spammish" or not. This is not a complete list, but I'll update it as I find more useful tools.


This is an IP clearinghouse run by IronPort, and used by many commercial anti-spam systems (like, Barracuda). You can check your IP or domain status here.

RBL checks

There are plenty of RBL systems out there, and plenty of tools to check your status. Here are a few:


Anti-Spam and email tools

This is a short list of tools I use to manage my mail servers (and mailboxes). The main building blocks for my system are
* [http://www.ogre.com|sendmail] for the MTA, with support for [http://www.milter.org|Milters]
* [http://asg.web.cmu.edu/cyrus/|Cyrus IMAP] server for IMAP over SSL
* Server side mail filters using [http://asg.web.cmu.edu/cyrus/sieve|Sieve]
* Webmail access using [http://www.squirrelmail.org/|SquirrelMail]
!!! sendmail
I'm running sendmail v8.13.1 currently, supporting a number of domains and users. sendmail uses the Cyrus mailer for delivery, which in turns talks to an [http://www.faqs.org/rfcs/rfc2033.html|LMTP] server. This injects the new messages directly into the IMAP folders. As an extra benefit, I also get the __+__ style addressing support automatically, meaning I can sort into sub-folders by the envelope address. For instance, if you send an email to


it is automatically sorted into my IMAP folder named __other.spam__.
!!! Cyrus IMAP
Cyrus has been my choise of IMAP server for quite a while, way back when there was really no comparison in features and performance. Netscape Mail server was very promising, but unfortunately died during the AOL and Sun massacres. Cyrus supports SASL of course, but also a somewhat unknown sub-system called [http://asg.web.cmu.edu/cyrus/sieve|Sieve]. This is a 100% server side mail filtering language, and it's quite powerful, much more so than I can document here. But, here's an example filter file:

require "fileinto";

if header :is "X-Spam-Flag" ["YES"] {
fileinto "INBOX.zSpam";
} elsif address :is :all ["from", "to", "cc", "bcc"] ["foo@ogre.com", "bar@ogre.com"] {
fileinto "INBOX.filtered.giants";
} elsif address :is :all ["to", "cc", "bcc"] ["dist-update@perldap.org"] {
fileinto "INBOX.filtered.dist";
} elsif header :contains "List-Id" "some_mail_list" {
fileinto "INBOX.mailists.games";

!!! Spam tools
I use a few sendmail milters to help get rid some of the worst spam:
* [http://www.mimedefang.org/|MIMEDefang]
* Snert's [http://www.milter.info/milter-spamc/index.shtml|milter-spamc]

I'm fully aware that MIMEdefang supports [http://www.spamassassin.org|SpamAsassasin], but I prefer to run these two systems independently. In fact, I wish MIMEdefang would make it easier to disable the user of SA! For SpamAssassin I use pretty much every bell and whistle there is, including:
* Pyzor
* Razor

!!! Anti-virus
Although MIMEDefang does some AV already, it also supports "pluggable" virus detectors. I currently use
* [http://www.clamav.net/|ClamAV]


sendmail, SASL, TLS and SSL

! Project goal
When I started this project, my primary goal was to allow for authenticated SMTP over a secure channel. I use [http://www.sendmail.org/|sendmail] as my MTA, not because I'm particularly fond of it, but because it's what I know, and it works well for me. My setup is pretty straight forward:
* ))RedHat(( linux (mix of 8.0 and 9.0)
* sendmail 8.12.10
* Latest OpenSSL and SASL libraries (I gave up on using RHs outdated builds)
* A myriad of milters, spam filters, and tools around sendmail and Cyrus IMAP.

I'm not going to go into details on the exact mail server configurations, but I'm concentrating on the changes I made to enable SSL and SASL for sendmail.
! Clients and TLS vs SSL-only
We mostly use well behaved mail clients for MUAs, but my wife insists on using Entourage on her Mac. To her defense, there's not a lot of alternatives out there for Macs, Mozilla isn't working very well on her system. Being a Microsoft client, Entourage of course doesn't follow the specifications. In particular, it doesn't support the __STARTTLS__ command. The client would simply complain that the server didn't provided any SASL authentication method that it could use:

The SMTP server for "Ogre" does not recognize any of the authentication
methods supported by Entourage. To send mail, try disabling SMTP
authentication in the account settings or talk to your administrator.

An unknown error (5530) occurred

The lack of proper SASL support in Entourage forced me to also include support for a dedicated SSL/TLS SMTP server (__SMTPS__). Fortunately, sendmail allows me to serve both the regular __SMTP__ port (25) and __SMTPS__ (port 465) using the same configuration.

As if this wasn't enough, once I got past this problem, Entourage misinterpreted the sendmail greetings as if client authentication (certificate) was required. I would get a client error like:

Security failure. Personal certificate required.

Fortunately sendmail provides a configuration solution for this as well, naturally. But seriously, why does MS have to make my life miserable all the time? I don't even hate them, I just prefer not to use most of their junk...

! Rebuilding sendmail with SASL support
This was the easy part, simply modify your __devtools/Site/site.config.m4__ file to something like:

APPENDDEF(`conf_sendmail_LIBS', `-lsasl2 -lssl -lcrypto')
APPENDDEF(`confLIBDIRS', `-L/usr/local/lib -R/usr/local/lib')
APPENDDEF(`confINCDIRS', `-I/usr/local/include')

APPENDDEF(`conf_sendmail_ENVDEF', `-D_FFR_SMTP_SSL')

You'll obviously have to change your directories etc. to match your build directory. The last configuration option enables an (experimental?) support for the SSL-only daemon support. I'm not 100% sure, but I think support for __SMTPS__ is enabled by default in __sendmail__ 8.13. In any case, this feature is important to work around the lack of proper TLS support in Entourage (which our new sendmail daemon will announce).

Make sure to save a copy of your old sendmail binary before installing this new build. Nothing worse that not being able to do a quick rollback on your oh so important mail servers.

! SSL certificates
I'm not going to go into detail on how to generate the appropriate SSL certificates here, there are plenty of sites that describes that. For example:
* [http://sapiens.wustl.edu/~sysmain/info/openssl/|OpenSSL HowTo]
* [http://www.pseudonym.org/ssl/ssl_cook.html|OpenSSL cookbook]

I personally just use my own root-CA, so I can avoid the Verisign tax. And this is plenty secure for all my needs.

! Reconfigure sendmail
First of all, we need to configure sendmail to find all the required certificate information. Assuming you are using the M4 macro packages for configuring sendmail, add something like this to your __sendmail.mc__:


Obviously, you must change the paths here. Next, we need to configure SASL properly, and here's the first stumbling block trying to support Entourage. As far as I can tell, you ''must'' have support for the __LOGIN__ authentication mechanism enabled. This might require a recompile of your SASL libraries, since it's an optional feature. My __sendmail.mc__ adds the following lines for SASL:

define(`confDONT_BLAME_SENDMAIL', `GroupReadableSASLDBFile')dnl

I only support "plain" SASL authentication, but other authentication mechanisms includes CRAM-MD5 and DIGEST-MD5. Using such mechanism requires you to setup and manage the SASL authentication database properly. The last configuration eliminates a sendmail runtime warning, if your SASL DB is group readable (common, to give sendmail read permissions?).

Entourage still won't talk to this sendmail configuration, because of it's broken STARTTLS implementation. The only workaround I know of is to enable the SSL-only port/daemon. This is done with a few more configuration lines to your __sendmail.mc__:

DAEMON_OPTIONS(`Port=smtp, Name=MTA')dnl
DAEMON_OPTIONS(`Port=465, Name=SSA, M=Eas')dnl
define(`confAUTH_OPTIONS', `A,p,y')dnl

The last option isn't absolutely necessary, but it prevents people from accidentally trying to use SMTP Authentication over a non-secure channel. It also denies anonymous logins. Alright, we're almost there. Now there's only one last problem, Entourage still complains about the requirement for certificate based authentication. Once again, sendmail provides configurations options for this:


If it hadn't been for Entourage (and I'm guessing, Outlook), enabling SASL and TLS for sendmail is pretty straight forward. I had it up and running for most of my clients in less than an hour, but figuring out all the tweaks for the rouge MUAs added several more hours of debugging and reconfigurations.

Is it worthwhile doing this? Absolutely! With SASL enabled (over TLS hopefully), your users can send email through your system from anywhere. One major benefit is that an authentication SMTP sessions is allowed to "gateway" mail just as if the client came from one of your blessed (internal) networks. And spammers can not abuse is, unless they can compromise your authentication mechanisms (i.e. hacking accounts).

Many thanks to my wife Michelle for using Entourage, and forcing me into fixing all this crud ... :-)

-- [mailto:leif@ogre.com?Subject=Re: ABIT IC7 audio for Linux|Leif Hedstrom]



Mozilla provides a couple of tools that I use on a regular basis.

Firefox performance vs compiler options

I've compiled the latest Firefox source (nightly) with various gcc compiler optimization options, compared to the "default" moz build options on my platform. This is not a comprehensive test, but merely gives some ideas of where the "best bang for the buck" is. I ran the Sunspider benchmarks three times for each build instance. My box is a single CPU core2 system with 4GB RAM. The compiler options used were as follow:

-Os -march=core2 -mtune=core2
-O2 -march=core2 -mtune=core2
-O3 -march=core2 -mtune=core2

The last is the default optimization options on my platform, which is what I call the "Moz" build. The "core2" options turns out to have little effect (1.01x - 1.03x at the most), but "-O3" vs "-O2" has significant impact ("-O3" always generates the fastest browser on my platform). And of course, the new JIT compiler is very fast. For now, all I have is this little screen shot of a table I made with the results, I'll convert this to a proper HTML table in my copious spare time.

The way to read this table is a little bit complicated, but pick a row (e.g. "O3 + JIT") and follow it to the column you wish to compare it to (e.g. Moz + JIT), and you see that the "O3 + JIT" build is 1.10x faster. The conclusions I can make is that it's always best to optimize as much as possible, optimizing for size doesn't makes sense on these modern CPUs (core2), for the Firefox code base. And JIT is always vastly superior.

As a side note, the build with the "-Os" options (optimize for size) was incredibly unstable with the JIT compiler enabled. It took me many, many runs to get it to complete three benchmarks without hanging.


Mobile and Portable devices

I really like my iPhone, and it has replaced my PSP as a portable media device.


Places to Get Ebooks

Amazon specific search - This search removes all the public domain books and sorts price from low to high.
Amazon Special Offers - All the Kindle special offers on Amazon
Manybooks - Kindle format for downloads. Foreign language books as well. Has public domain books too.
Manybooks (Kindle edition) - Access through the Kindle web browser to download directly to Kindle. Choose Mobipocket format.
Feedbooks - Kindle format for downloads. Has public domain books too and I think others.
Project Gutenberg - Multiple languages. Seems to be in HTML or Text format (download text for the Kindle)
Free Kindle Books - a bunch of Project Gutenberg books in Kindle format.
Fictionwise - .mobi format. Some books free, some not, but some sales as well on the non-free ones.
World Public Library - all PDFs. Cost is $8.95/year. Over 400,000 classic titles.
Fictionpress - mostly original works, in Text format.

Conversion Software

Calibre - converts to/from a bunch of different formats including PDF and Kindle. Works on Windows, Mac, Linux.
MobiPocket - converts from PDF/Word/Text to .prc which can be read by the Kindle. Works on Windows.
- export stuff to Kindle format. Probably others, but not sure which. Works on Windows, Mac.



Here's a list of interesting and upcoming conferences.

  • Usenix LISA '12 (proposals due May?)
  • http://www.dynamic-languages-symposium.org/dls-12/cfp/index.html

HTTP Tools

This is a collection of links and info on HTTP related tools.

This is work in progress...


There's a number of benchmarking tools

  • http_load or the beefed up version that comes with ATS
  • vegeta is a nice tool written in the Go language
    • Install: $ go get -u github.com/tsenart/vegeta
  • wrk a fast, multithreaded load-tester, written in C.

Other test tools

More random test tools, useful for debugging and other tests.

  • httplab is an awesome little tool to simulate origin server behavior.
    • Install: $ go install github.com/gchaincl/httplab/cmd/httplab



Small collection of some useful network and TCP tidbits.

TCP socket options

Some collected useful tips about various TCP socket options, e.g. setsockotp() and getsockopt().


This is availble on Linux, and is a poor mans "accept filtering" (as FreeBSD implements it). Apache HTTPD v2.2.x sets this to "1" (which I assume was a mistake), which translates to 3s before timing out. HTTPD 2.3.x increases this to 30s, which according to the table I found for Linux v2.6.32 (and later) translates to 45s. Internally, the kernel translates the timeout to a number of retransmits to perform before the timeout, which means a call to setsockopt() followed by getsockopt() won't necessarily show the same value. I found the following conversions, from the requested timeout (in seconds) to effective timeout as used internally):

1-3	-> 3
4-9	-> 9
10-21	-> 21
22-45	-> 45
46-93	-> 93
94-189	-> 189
190-309	-> 309
310-429	-> 429

On older kernels, I see the following conversion (due to different rounding I think):

1-3	-> 3
4-6	-> 6
7-12	-> 12
13-24	-> 24
25-48	-> 48
49-96	-> 96
97-192	-> 192
193-384	-> 384

The code for this is available here.