New Python features, and super()
I'm a long time Python user (since 1994) and fan, and lately I've been trying to catch up on some of the many changes the language have been going through. I must say, pretty much everything they have changed or added since the old 1.x days are awesome! As if Python wasn't a great language already, it's shaping up to be a very strong contender indeed.
If you haven't looked at Python lately, take a quick look at the following "summaries" of changes that's been made in recent years:
- What's new in python 2.0
- What's new in Python 2.1
- What's new in Python 2.2
- What's new in Python 2.3
- What's new in Python 2.4
- What's new in the next Python
- Python PEPs
Anyways, last week I was playing with the super() function, to clean up some old code which used the old style calling conventions for accessing the super class members. This new built-in function was added to Python 2.2, to better support the new multiple inheritance lookup rules (see above). The following example shows the two different styles (but read the docs above for more in depth analysis why using super() is useful):
class Foo(BaseFoo):
def __init__(self, arg1, arg2):
BaseFoo.__init__(self, arg1, arg2)
...
class Bar(BaseBar):
def __init__(self, arg1, arg2):
super(Bar, self).__init__(arg1, arg2)
...
This worked most of the time in my code, except when I tried to sub-class some old-style classes, for instance:
from HTMLParser import HTMLParser, HTMLParseError
class HTMLImageParser(HTMLParser):
def __init__(self, callback=None):
super(HTMLImageParser, self).__init__()
....
This would generate an error like:
TypeError: super() argument 1 must be type, not classobj
The simple solution was to change my sub-class to also be a sub-class of the object class, making it a new-style class:
class HTMLImageParser(HTMLParser, object):
def __init__(self, callback=None):
super(HTMLImageParser, self).__init__()
...
sendmail, SASL, TLS and SSL
Project goal
When I started this project, my primary goal was to allow for authenticated SMTP over a secure channel. I use sendmail as my MTA, not because I'm particularly fond of it, but because it's what I know, and it works well for me. My setup is pretty straight forward:
- ))RedHat(( linux (mix of 8.0 and 9.0)
- sendmail 8.12.10
- Latest OpenSSL and SASL libraries (I gave up on using RHs outdated builds)
- A myriad of milters, spam filters, and tools around sendmail and Cyrus IMAP.
I'm not going to go into details on the exact mail server configurations, but I'm concentrating on the changes I made to enable SSL and SASL for sendmail.
Clients and TLS vs SSL-only
We mostly use well behaved mail clients for MUAs, but my wife insists on using Entourage on her Mac. To her defense, there's not a lot of alternatives out there for Macs, Mozilla isn't working very well on her system. Being a Microsoft client, Entourage of course doesn't follow the specifications. In particular, it doesn't support the STARTTLS command. The client would simply complain that the server didn't provided any SASL authentication method that it could use:
The SMTP server for "Ogre" does not recognize any of the authentication
methods supported by Entourage. To send mail, try disabling SMTP
authentication in the account settings or talk to your administrator.
An unknown error (5530) occurred
The lack of proper SASL support in Entourage forced me to also include support for a dedicated SSL/TLS SMTP server (SMTPS). Fortunately, sendmail allows me to serve both the regular SMTP port (25) and SMTPS (port 465) using the same configuration.
As if this wasn't enough, once I got past this problem, Entourage misinterpreted the sendmail greetings as if client authentication (certificate) was required. I would get a client error like:
Security failure. Personal certificate required.
Fortunately sendmail provides a configuration solution for this as well, naturally. But seriously, why does MS have to make my life miserable all the time? I don't even hate them, I just prefer not to use most of their junk...
Rebuilding sendmail with SASL support
This was the easy part, simply modify your devtools/Site/site.config.m4 file to something like:
APPENDDEF(`confENVDEF', `-DSASL=2 -DSTARTTLS')
APPENDDEF(`conf_sendmail_LIBS', `-lsasl2 -lssl -lcrypto')
APPENDDEF(`confLIBDIRS', `-L/usr/local/lib -R/usr/local/lib')
APPENDDEF(`confINCDIRS', `-I/usr/local/include')
APPENDDEF(`conf_sendmail_ENVDEF', `-D_FFR_SMTP_SSL')
You'll obviously have to change your directories etc. to match your build directory. The last configuration option enables an (experimental?) support for the SSL-only daemon support. I'm not 100% sure, but I think support for SMTPS is enabled by default in sendmail 8.13. In any case, this feature is important to work around the lack of proper TLS support in Entourage (which our new sendmail daemon will announce).
Make sure to save a copy of your old sendmail binary before installing this new build. Nothing worse that not being able to do a quick rollback on your oh so important mail servers.
SSL certificates
I'm not going to go into detail on how to generate the appropriate SSL certificates here, there are plenty of sites that describes that. For example:
I personally just use my own root-CA, so I can avoid the Verisign tax. And this is plenty secure for all my needs.
Reconfigure sendmail
First of all, we need to configure sendmail to find all the required certificate information. Assuming you are using the M4 macro packages for configuring sendmail, add something like this to your sendmail.mc:
define(`confCACERT_PATH',`/usr/local/SSL/private')dnl
define(`confCACERT',`/usr/local/SSL/private/CAcert.pem')dnl
define(`confSERVER_CERT',`/usr/local/SSL/certs/sendmail-cert.pem')dnl
define(`confSERVER_KEY',`/usr/local/SSL/certs/sendmail-key.pem')dnl
Obviously, you must change the paths here. Next, we need to configure SASL properly, and here's the first stumbling block trying to support Entourage. As far as I can tell, you must have support for the LOGIN authentication mechanism enabled. This might require a recompile of your SASL libraries, since it's an optional feature. My sendmail.mc adds the following lines for SASL:
define(`confAUTH_MECHANISMS',`LOGIN PLAIN')dnl
TRUST_AUTH_MECH(`LOGIN PLAIN)dnl
define(`confDONT_BLAME_SENDMAIL', `GroupReadableSASLDBFile')dnl
I only support "plain" SASL authentication, but other authentication mechanisms includes CRAM-MD5 and DIGEST-MD5. Using such mechanism requires you to setup and manage the SASL authentication database properly. The last configuration eliminates a sendmail runtime warning, if your SASL DB is group readable (common, to give sendmail read permissions?).
Entourage still won't talk to this sendmail configuration, because of it's broken STARTTLS implementation. The only workaround I know of is to enable the SSL-only port/daemon. This is done with a few more configuration lines to your sendmail.mc:
DAEMON_OPTIONS(`Port=smtp, Name=MTA')dnl
DAEMON_OPTIONS(`Port=465, Name=SSA, M=Eas')dnl
define(`confAUTH_OPTIONS', `A,p,y')dnl
The last option isn't absolutely necessary, but it prevents people from accidentally trying to use SMTP Authentication over a non-secure channel. It also denies anonymous logins. Alright, we're almost there. Now there's only one last problem, Entourage still complains about the requirement for certificate based authentication. Once again, sendmail provides configurations options for this:
define(`confTLS_SRV_OPTIONS',`V')dnl
Conclusions
If it hadn't been for Entourage (and I'm guessing, Outlook), enabling SASL and TLS for sendmail is pretty straight forward. I had it up and running for most of my clients in less than an hour, but figuring out all the tweaks for the rouge MUAs added several more hours of debugging and reconfigurations.
Is it worthwhile doing this? Absolutely! With SASL enabled (over TLS hopefully), your users can send email through your system from anywhere. One major benefit is that an authentication SMTP sessions is allowed to "gateway" mail just as if the client came from one of your blessed (internal) networks. And spammers can not abuse is, unless they can compromise your authentication mechanisms (i.e. hacking accounts).
Credits
Many thanks to my wife Michelle for using Entourage, and forcing me into fixing all this crud ... :-) -- Leif Hedstrom
MacOSX and DHCP LDAP option...
So, for the second or third time, I read on Slashdot about people complaining about the "serious security" problems with MacOSX. I finally couldn't stop myself from posting to one of these discussions, not because I'm a Macintosh fan, but because I find it prepostoreous to even call this a bug.
As far as I'm concerned, Apple has implemented their solutions as per specifications. They probably could have made better documentation, GUI/tools and training around the well known security issues around DHCP in general, and the LDAP option in particular. And arguably, using DHCP with the LDAP option like they did might have been a poor design decision, but it was no less secure than their previous systems afaik (e.g. Netinfo).
I first posted this response to the latest /. article:
I still don't understand why this security "hole" got so much attention... Are people struggling
to find problems with MacOSX? First of all, attacks like this is nothing new, just remember the
old YP/NIS problems with broadcasting for the server, to mention just one example.
Secondly, when we wrote the DHCP LDAP option specs way back when, we explicitly
documented this problem in the security section:
5. Security considerations
Security considerations discussed in [3], particularly with respect to the
provision of authentication information, are directly applicable here.
Additionally, it should be noted that providing LDAP server information by
a broadcast protocol such as DHCP may allow unauthorized clients to learn
the location of and authentication information for LDAP servers and hence
pose as valid clients. This presents a security problem when sensitive
information, such as user passwords, is published via LDAP servers.
The DHCP protocol provides no mechanisms for the client to verify the
validity and correctness of the received information. The security
considerations in [1] discuss several weaknesses, particularly the problem
with unauthorized DHCP servers.
This was written in 1997, note the last paragraph above. These issues have been discusses
and documented in several RFCs, many years ago...
Someone posted a question asking if this was a MacOSX specific problem, to which I responded:
Well, DHCP is inherently insecure, so this is definitely not a MacOSX specific "bug" (but I
personally don't consider it a bug). This is all well documented in the DHCP RFCs and docs,
e.g. from RFC 2131:
7. Security Considerations
DHCP is built directly on UDP and IP which are as yet inherently
insecure. Furthermore, DHCP is generally intended to make
maintenance of remote and/or diskless hosts easier. While perhaps
not impossible, configuring such hosts with passwords or keys may be
difficult and inconvenient. Therefore, DHCP in its current form is
quite insecure.
Unauthorized DHCP servers may be easily set up. Such servers can
then send false and potentially disruptive information to clients
such as incorrect or duplicate IP addresses, incorrect routing
information (including spoof routers, etc.), incorrect domain
nameserver addresses (such as spoof nameservers), and so on.
Clearly, once this seed information is in place, an attacker can
further compromise affected systems.
Malicious DHCP clients could masquerade as legitimate clients and
retrieve information intended for those legitimate clients. Where
dynamic allocation of resources is used, a malicious client could
claim all resources for itself, thereby denying resources to
legitimate clients.
I think what makes MacOSX "unique" is that they use services traditionally not provided by DHCP (in this case, LDAP server information). Just like with NIS/YP, we have a tradeoff between ease of deployment (automatic service discovery) vs strong security. I know for a fact that way back, many YP/NIS deployments got hacked (in open networks, most commonly Universities) by simply pretending to be an NIS server. NIS+ addressed this problem (and others), and made it close to impossible to deploy and maintain. :-)
I don't know what Apple will do to "secure" this, the natural solution seems to be to have the DHCP client limit which servers it will talk to (establish a trust relation). It could be done with something as simple as a DHCP server host list, or more likely using Kerberos tickets to verify the authenticity of the DHCP response (I'm no Kerberos expert, so don't quote me on that one). More than likely, it'll make deployment a bit harder that what it is now.
Nothing new about this at all....
Spam...
Some readers might know that one of my pet peeves is Spam. I've spent so much time trying to fight spam, and I'm actually "winning" right now. It is a constant battle though, as spammers adapt to new (and old) anti-spam tools, and I have to constantly update and tune my systems.
I guess one of these days I should write something about what I do for anti-spam, I just haven't had the time. I do run a number of different toolkits together, which has turned out to be very effective. A few months ago I added a Bayesian filter system, which even though it is effective, is incredibly painful to maintain. I'm also thinking of adding RBL to my systems, assuming they let me use it for free. :-)
I'm working on a tool to let me manage the learning through IMAP clients (e.g. Mozilla), more or less automatically. I'll post more on that when I get closer to release. (And yes, I know there are similar tools out there already, but I need something that's 100% server side).
Oh, and for what it's worth, I'm currently catching 10k spam / month / user on my system, and that's not counting spam/virus that are rejected immediately at the MTA layer. That'd add up to well over 100k spam / year for one person, it's just insane. For me personally, spam is over 75% of my incoming mail, and I'm on a lot of fairly large mailing lists...
Stop spam
VMWare, RedHat 9 and Linux 2.6
---
UPDATE!
This seems to have been resolved in the latest vmware "fix" package from Petr (v44), see more info in this Bugzilla bug.
---
UPDATE #2!
I think they removed this compile option again from the unofficial 2.6 builds, maybe because it broke other binary only kernel modules (like nVidia)? But the nVidia drivers are still broken, since the new CONFIG_X86_4G_VM_LAYOUT is used. More information on the nVidia for 2.6 site.
---
So, I've been tooling around using the 2.6 RPMs provided by some nice fellow at RedHat . I simply followed the instructions from Thomer M. Gil. It was suprisingly easy to get my ))RedHat 9(( system upgraded to 2.6 using these RPMs and instructions.
This all worked nice and well up until (I think) test-8. After this release, I could no longer get VMWare to run, even after applying Petr's patches. Reading the News group for running VMWare on "experimental plaforms", I found this comment from Petr:
^''linux-2.6.0-compile.patch in Arjan's kernel is responsible for that. It adds __-mregparm=3__ to the CFLAGS, completely breaking ABI: functions do not expect arguments on the stack, but in registers (and so misc_register() uses leftover 165 value in EAX as a miscdevice pointer...).''^
I personally "solved" this by editing my arch/i386/Makefile in the Kernel source, removed the -mregparm=3 option, and rebuilt the kernel.
Ximian "devel" packages and Emacs
I've been playing around a bit with the latest "unstable" Ximian Gnome packages, with mixed results. Most disturbing, my Emacs settings got completely whacky. My colors were all changed, fonts weren't right, and many other of my custom settings were being ignored. After some annoying debugging, I realized Gnome was now setting some X11 resources, which obviously affected my Emacs run-time.
The offending file (at least for me), turned out to be:
/usr/share/control-center-2.0/xrdb/Emacs.ad
which is part of the control-center package. Simply removing this file, and restarting Gnome/X11 solved all my Emacs problems, since the conflicting X11 resources are now gone. I have no idea why this file was put in there... I control all my Emacs settings through E-lisp, thank you very much, and X11 resources should be avoided when possible IMO. :-)
ABIT IC7 audio for Linux
Introduction
To begin with, it seems the native RedHat/Linux driver for the Intel 8x0 audio devices does not recognize the on-board audio chip on ABIT's motherboard. I did some research, and my web searches and documentation reading didn't give any positive results (although, others have reported that similar systems worked out of the box with the RH9 drivers). I'm guessing it's something to do with how the ABIT card is registering the audio device, like
00:1f.5 Multimedia audio controller: Intel Corp. 82801EB AC'97 Audio (rev 02)
Subsystem: ABIT Computer Corp.: Unknown device 1022
Flags: bus master, medium devsel, latency 0, IRQ 17
I/O ports at d400 [size=256]
I/O ports at d800 [size=64]
Memory at fc002000 (32-bit, non-prefetchable) [size=512]
Memory at fc003000 (32-bit, non-prefetchable) [size=256]
Capabilities: [50] Power Management version 2
It's quite possible you can get the 8x0 OSS driver to work on your system, but I wasn't in the mood to mess with that, so I gave up.
ALSA to the rescue!
After ditching the "native" 8x0 driver provided by ))RedHat((, I went to download and build ALSA (The Advanced Linux Sound Architecture) instead. However, to save some time, it's much easier to find some existing RPMs and simply install those. For instance, FreshRPMs provide an excellent library of packages built for ))RedHat(( 9. I installed the following RPMs on my system:
kernel-smp-module-alsa-0.9.6-1.fr_2.4.20_20.9
gnome-alsamixer-0.9.3-fr1
alsaplayer-0.99.75-fr1
alsa-lib-0.9.6-1.fr
alsa-utils-0.9.6-1.fr
xmms-alsa-0.9.12-1.fr
alsa-driver-0.9.6-1.fr
Some of these packages depend on other components, but it's pretty straight forward to figure out which ones you will need. Make sure you pick the right kernel modules for ALSA, matching your existing kernel as installed on your system!
If you play MP3's while hacking along, like I do, getting the ALSA output plugin for XMMS is a must. It was straight forward to get it going, just change your XMMS to use the ALSA output plugin, and you should be ready to go.
Could this work with the ABIT IC7?
After rebooting the system, Kudzu quickly discovered my new audio device, and configured it. Yes, I finally have audio!?! Not so fast my young padawan, Gnome was still very unhappy, alsaplayer, xmms and other audio players were still all painfully silent. Have I ever mentioned I hate computers sometimes?
Not ready to give up quite yet, I fired up the alsamixer, and I noticed pretty much all of the channels were muted by default. "That can't be good!", I said to myself, so I unmuted the relevant channels, and adjusted the sound levels appropriately. Holy cow, there be sound! And the sound was good!
Once I configured the ALSA mixer properly, I made sure to save the new default configuration (to /etc, with root privs), using the alsactl command:
root # alsactl store
To assure that sound is restored properly when rebooting the system, I added a couple of lines to my /etc/modules.conf file:
alias sound-slot-0 snd-intel8x0
post-install snd-slot-0 /usr/sbin/alsactl restore >/dev/null 2>&1 || :
But what about the Gnomes?
I was still out of luck getting the Gnomie to produce any sounds at all, not even a whimper. Esound (and the esd server) was unhappy about not finding the appropriate /dev devices to play with. What's worse, there was no sound while playing my current favorite game on Linux, Neverwinter Nights. It was getting really late now, but not having sound while cleansing the plague from Neverwinter simply wasn't acceptable.
More searches on Yahoo and Google indicated that ALSA should have an OSS compatibility mode. That looked promising, so I read through more ALSA docs, and found an interesting tidbit. Adding a few more lines to modules.conf should supposedly do the trick:
alias sound-service-0-0 snd-mixer-oss
alias sound-service-0-1 snd-seq-oss
alias sound-service-0-3 snd-pcm-oss
alias sound-service-0-8 snd-seq-oss
alias sound-service-0-12 snd-pcm-oss
A quick reboot, god I love my new 3GHz P4 machine, and I have a happy Gnome again. Now for the final test, will NWN produce sound? Yes!!! I'm back in business, my Paladin can once again rid the world of all that is evil, and the peasants can rejoice.
Conclusions
I must say the Linux world still has some ways to go when it comes to audio support. It shouldn't have to be this difficult to get sound working, and I don't think I should have to pay extra money to get an audio driver that works (which I have done in the past).
ALSA seems to work extremely well so far, and I'm very pleased with it. Enabling the OSS compatibility mode was really easy, and I got the best of both worlds. Way to go ALSA people, I might even have to switch to SUSE Linux now.
Updates and user feedback
11/22/2003: XMMS and ESD
Fredrik S. Bredeli sent me an email informing me that using xmms with ESD The (Enlightened Sound Daemon) solved audio resource contention issues for him. Just make sure the ESD daemon is running properly, easiest using the Gnome configuration tools. Or, manually with something like
esd -d /dev/dsp
Once ESD is running, just switch XMMS to use the eSound Output Plugin again.
---
-- Leif Hedstrom