The Fugue

I have a 64-bit desktop machine, that has rarely been run as a 64-bit machine. The hassle was too great and I couldn't really see a reason to put up with it.

I think that 64-bit support has come a long way in the meantime, and it may be time to try it out. It sounds like a livable situation. So with the pending release of the next Ubuntu version I'm thinking of wiping and going 64-bit.

One of the primary motivators is that 64-bit holds some promise for transcoding video, and now that I have an HDHomeRun to capture over-the-air HDTV signals, I will be doing quite a bit of video transcoding for MythTV (to save disk space—a full-quality HDTV program is about 9 gigabytes per hour).

But before taking the plunge, I thought I'd do an empirical test and see if there would be any real savings. I captured a couple of minutes of HD content from PBS, then transcoded 60 seconds using ffmpeg and mencoder. Then I did the same with the Ubuntu 64-bit live CD. The 64-bit execution difference was statistically significant.

ffmpeg was about 1.12 times as fast—a savings of about 10 seconds per minute, or 10 minutes per hour.

mencoder was about 1.08 times as fast—similar savings.

I didn't test mythtranscode itself, since getting it installed in a live CD environment would be too much work. I also must point out some other possible confounding variables. I used the Ubuntu 7.10 versions of ffmpeg and mencoder in 32-bit, and the Ubuntu 8.04 versions in 64-bit. Did both projects improve their code to be about 10% faster in the meantime? Unlikely, but perhaps not unfathomable.

So will I make the switch? I don't know yet. 10% faster is significant, but not obviously worth it. I'll have to think about it.

For the curious, here's my numbers. I did at least two runs of each to check for agreement, and what you see is the average. Of course, these would not be the settings you'd necessarily use to transcode—ffmpeg has a pretty low default bitrate for example—but I think we can agree the speedup is likely to be in the same ballpark no matter what settings you're using.

# 64-bit    32-bit
# 86 s      95 s
time ffmpeg -y -t 60 -i foo.avi -acodec copy bar.avi
# 55 s      64 s
time ffmpeg -y -t 60 -i foo.avi -acodec copy -s 640x480 bar.avi
# 83 s      90 s
time mencoder foo.avi -oac copy -ovc lavc -frames $[30*60] -o baz.avi

Postfix is my MTA of choice. Recently I had a second opportunity to set up relaying from Postfix to Postfix, with TLS and authorization. Seeing how I remembered precious little from the first time, I decided it would be a good thing to blog on.

The documentation on doing this is really quite good, but you have to get acclimated to the acronym soup before it makes any sense at all. The first and most mysterious acronym is SASL.

SASL is the Simple Authentication and Security Layer, a method for adding authentication support to connection-based protocols. To use SASL, a protocol includes a command for identifying and authenticating a user to a server and for optionally negotiating protection of subsequent protocol interactions. If its use is negotiated, a security layer is inserted between the protocol and the connection.

If that's not a mouthful… Basically, SASL is a library and daemon that programs, like Postfix, can use to do authentication. The Postfix SASL Howto tells you all you need to know about configuring Postfix for Cyrus or Dovecot SASL. It also tells you how to configure either Dovecot or Cyrus SASL for Postfix.

I'm using Debian stable (4.0), and this is what I did. On both the client and server you need the postfix-tls package which includes SASL and TLS support for Postfix. On the server I had to install the sasl2-bin package (this is not at all obvious at first pass—I was looking for a saslauthd package). Then I had to enable saslauthd by editing /etc/default/saslauthd. The smtpd.conf file is in /etc/postfix/sasl on Debian, and it looks like this:

pwcheck_method: saslauthd
mech_list: PLAIN LOGIN

Here's the relevant snippet from /etc/postfix/main.cf:

smtpd_sasl_auth_enable = yes
smtpd_sasl_application_name = smtpd
smtpd_sasl_security_options = noanonymous
smtpd_recipient_restrictions = 
    permit_mynetworks
    permit_sasl_authenticated
    reject_unauth_destination

Now, there's a problem. Debian runs Postfix in a chroot jail by default, which means you need to make special provision for Postfix to be able to find the saslauthd socket. This can be as easy as

mv /var/run/saslauthd/ /var/spool/postfix/var/run/
ln -s /var/spool/postfix/var/run/saslauthd/ /var/run/

You may also need to adduser postfix sasl, though I'm not sure if this is necessary.

That's it for the server. Now, on the client you need this in /etc/postfix/main.cf:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
relayhost = [mail.example.com]
smtp_sasl_security_options = noanonymous

/etc/postfix/sasl_passwd looks like this:

[mail.example.com] username:password

You need to postmap /etc/postfix/sasl_passwd after changing it.

Now, authentication is well and good, but you don't want to be sending those passwords in the clear, especially when using the default PAM authentication source. So, you also need to configure TLS.

The Postfix TLS README tells you all you need to know for this. You need to create a certificate for the server, enable the use of TLS on both sides, and tell the server not to accept authentication without TLS. That last bit is perhaps the most vital element for security, though of course it does nothing to help you get TLS actually working. Here's the config snippet:

# client
smtp_use_tls = yes # This option deprecated in later versions of Postfix

# server
smtpd_tls_CAfile = /etc/ssl/CA/cacert.pem
smtpd_tls_cert_file = /etc/ssl/certs/mail.pem
smtpd_tls_key_file = /etc/ssl/private/mail.pem
tls_random_source = dev:/dev/urandom
smtpd_tls_loglevel = 1
smtpd_use_tls = yes # also deprecated

Creating the certificates is nothing extraordinary, but this seems like a good time to post my /etc/ssl/README file:

self-signed:
    openssl req -new -nodes -out newreq.pem
    openssl x509 -req -signkey privkey.pem -in newreq.pem -out cert.pem

create CA:
openssl req -nodes -new -x509 -days 3650
    -keyout CA/private/cakey.pem -out CA/cacert.pem

generate request:
openssl req -new -text -nodes -keyout newkey.pem -out newreq.pem

sign request:
openssl ca -in newreq.pem -out newcert.pem -days $((365*2))

Don't forget to keep private keys private.

So there it is. Authentication and Encryption at your fingertips.

I had the unfortunate occasion of reconfiguring my OpenWRT router after a fresh reflashing yesterday, without sufficient notes of how I had it set up. So, I'm documenting it this time. But it's not just notes for myself—I'm doing a few tricks that my dear readers might learn from.

First, the topology. I have a /29 subnet: .105–.110 (6 hosts). There is a Cisco DSL modem, a Buffalo router (running OpenWRT), my main server named falcon, a Sipura ATA, a desktop, and my laptop. The latter 4 are collectively the public-IP LAN, and the topology looks like this:

(internet) -- cisco -- buffalo +- public-IP LAN
                               |
                               +- private-IP LAN

The private-IP LAN is 172.17.77.0/24, and primarily for guest wireless devices or whatever other devices I happen to want to plug in.

The buffalo has one internal switch, to which is connected the wireless radio and 5 ethernet ports. It is split into 2 VLANs. vlan1 is the port connected to the cisco (.105), and the other ports and the wireless are bridged as br0 (with a private IP: 77.2).

Now remember, The cisco, buffalo, and public-IP LAN are all on the same /29 subnet. But the cisco and the public-IP LAN are on different VLANs. Let that sink in. You should be objecting about now, and questioning my sanity. (If you're not familiar with the term VLAN, it basically is a way to partition the broadcast domain.) How can I have two independent broadcast segments for one subnet?

Here's what the routing on the buffalo looks like:

# ip route
216.31.27.105 dev vlan1  scope link 
216.31.27.104/29 dev br0  scope link 
172.17.77.0/24 dev br0  proto kernel  scope link  src 172.17.77.2 
default via 216.31.27.105 dev vlan1

and this is how it is achieved:

# cat /etc/init.d/S41routing 
ip route del default
ip route del 216.31.27.104/29 dev vlan1
ip route add 216.31.27.104/29 dev br0
ip route add 216.31.27.105 dev vlan1
ip route add default via 216.31.27.105

So everything in .104/29 goes to the right, and only traffic to the cisco goes to the left. Of course the private subnet goes to the right too. The default route is via the cisco. (Incidentally, you need to install iproute2 for using ip: ipkg install ip)

You also need to configure a static route on the cisco:

set route add ip 216.31.27.104 netmask 255.255.255.248 gw 216.31.27.109

Remember though that the left-hand IP of buffalo is 216.31.27.109, and the right-hand IP is 172.17.77.2. So if falcon, which is on the right, tries to reach .109 (its default route), will it get an ARP response? The answer is, it depends. It depends on the effective arp_ignore sysctl setting. By default on OpenWRT 0.9 (White Russian), /proc/sys/net/ipv4/conf/{default,all}/arp_ignore is set to 1. This means "reply only if the target IP address is local address configured on the incoming interface", which means buffalo won't reply to ARP requests for .109 coming from the right. We need to set arp_ignore to 0 for that interface, meaning "reply for any local target IP address, configured on any interface." How do we do that?

"The max value from conf/{all,$interface}/arp_ignore is used when ARP request is received on the $interface." So, we ned to set conf/{all,br0}/arp_ignore to 0. We do this in /etc/sysctl.conf:

--- etc.orig/sysctl.conf        2007-11-24 08:30:22.000000000 -0700
+++ etc/sysctl.conf     2007-11-24 08:30:29.000000000 -0700
@@ -1,6 +1,7 @@
 kernel.panic=3
 net.ipv4.conf.default.arp_ignore=1
-net.ipv4.conf.all.arp_ignore=1
+net.ipv4.conf.all.arp_ignore=0
+net.ipv4.conf.br0.arp_ignore=0
 net.ipv4.ip_forward=1                 
 net.ipv4.icmp_echo_ignore_broadcasts=1      
 net.ipv4.icmp_ignore_bogus_error_responses=1

There, now (after reboot) the right side of buffalo can ping .109 (don't forget to clear the ARP cache of the host you're testing from), and therefore use .109 as a default route. Everything works great, with one exception. The right side of buffalo cannot ping the cisco. If that bothers you, you can add a static route to the hosts on the right side, like this:

ip route add 216.31.27.105 via 216.31.27.109

But in practice I don't bother. As long as you remember that you won't be able to and don't try to verify routing by pinging the cisco (pick an external IP address instead), you're fine. I can always ssh to buffalo and then have full access to cisco, or just use the serial connection from falcon.

Ok, so that's how I have one VLAN span two broadcast segments. That's not the only thing I do differently, though.

I disable DHCP on buffalo, because falcon is serving DHCP already. But I don't want to disable dnsmasq altogether because the dns bit is still useful, so I edit /etc/dnsmasq.conf and comment out dhcp-authoritative.

I also add this QoS script as /etc/init.d/S42qos.It is the excellent Wondershaper modified for the limitations of the OpenWRT environment. See the comments at the top of the file for instructions. With this QoS setup I don't need to throttle my bittorrent downloads or uploads, but am still able to make clear VOIP calls.

I don't like the default firewall (it's no good for a server behind it), and I find modifying its config is annoying, so I generate a fresh config using ferm and copy it over to buffalo as /etc/firewall.user.

Everything else that I have changed is fairly standard and doesn't need special description.

I love OpenVPN, but man what a man page. I've always been annoyed that I have to go digging through the manpage to figure out which options I need and what parameters they take, just to do stuff I already know how to do with iproute2, no doubt familiar to you all from your dutiful study of the LARTC. Right? Right?!

Finally I got fed up with it. I took a step back and looked at the big picture. There's 3 big pieces to this puzzle. First is tun/tap, which allows a userspace program to read and write IP or ethernet frames to a virtual network device. One could set up a virtual unprivate network using tun/tap. OpenVPN sets up a virtual private network, by adding all the authentication and encryption stuff. The third piece of the puzzle is the piece that's always there whenever you have any kind of network device, virtual or not. That's configuring the device and setting up the routing. OpenVPN has a plethora of options for all this, but IMHO it's better done with the proper tools, in good UNIX style.

So I changed up my OpenVPN server config to look like this:

# tap because we act like a switch
dev tap

# mode server and tls-server to allow us to have many tls clients
mode server
tls-server

# certificate stuff
ca falcon-cacert.pem
cert falcon-cert.pem
key falcon-key.pem
dh dh2048.pem

# I don't like openvpn's --ifconfig and --route family of options for my
# server, so I do it with a script instead
up "/etc/openvpn/falcon.up"

# allow clients to find other clients and their subnets with this pushed
# route
push "route-gateway 172.17.0.3"
push "route 172.17.0.0 255.255.0.0"
client-config-dir clients

# eventually it would be nice to use a real dhcp server instead of this
ifconfig-pool 172.17.0.200 172.17.0.250 255.255.255.0

# options
keepalive 10 60
comp-lzo
learn-address "echo $*"
status /var/log/openvpn.status 60

Let's walk through it. I use dev tap because I've never had occasion to notice the overhead and it makes wrapping your head around the routing a lot easier. Basically (along with a couple of bits later), the server acts as a switch into which all the clients plug. mode server and tls-server denote server mode (with public key authentication), instead of peer-to-peer (one-to-one) mode. We don't use server-bridge because that sets some options I want to take care of myself. The cert stuff is nothing new.

The next line is the important bit. We use an up script to do the equivalent of OpenVPN's ifconfig and route commands. Here is where the added flexibility and tool familiarity comes into play. Here's my up script:

#!/bin/sh

ip addr add 172.17.0.3/24 dev $dev
ip link set $dev up
ip link set $dev mtu $tun_mtu

for i in 64 77 79 82 84; do
    ip route add 172.17.$i.0/24 via 172.17.0.$i
done

When this script is run, $dev is tap0. I add the address and set the link up and set the mtu for the link. This is the equivalent of the OpenVPN ifconfig 172.17.0.3 255.255.255.0 which is implied by server-bridge with the appropriate options. The next bit is the equivalent of several route options in OpenVPN. Note how I stay DRY while adding 5 routes. If I was doing something more fancy, this script would allow me to do it in the natural way using the appropriate tools for the job.

Back to the OpenVPN config. I'm using push instead of putting the burden on future me or whoever is going to set up new clients down the road. In this case route and ifconfig-push (in the client configuration file) are the right tools for the job, and we're glad they're there. As an aside, notice the routes I'm setting up. The route-gateway 172.17.0.3 tells them where the "switch" is, and the route 172.17.0.0 255.255.0.0 tells them how to find other subnets. A quick example: ungol (172.17.0.64) and caradhras (172.17.0.82) to falcon (172.17.0.3). ungol's LAN subnet is 172.17.64.0/24. caradhras' LAN subnet is 172.17.82.0/24. If 172.17.64.x pings 172.17.82.x, these very important routes come into play. Otherwise, you'd only be able to reach falcon from either subnet. Naturally, the subnets need the correct routing too (ungol and caradhras are probably the default route for those subnets anyway). This is roughly the same thing that happens when you do server-bridge and client-to-client, but not precisely. The reverse routes (in the up script) are needed in either case. Incidentally, you could put client-to-client here too, which would keep the packets from leaving OpenVPN into the kernel stack halfway through their journey. I'm not sure how much overhead that is, but in the interest of minimalism (i.e. "because I can") I left it out.

The ifconfig-pool bit is playing at being a DHCP server. I'm even less happy about OpenVPN playing DHCP server than I am about ifconfig and route. I think configuring my DHCP server on falcon to serve up addresses and options instead of OpenVPN is doable, but there may be some timing issues and I still need to figure out the best way to trigger the DHCP behavior on the client. I just haven't tackled it yet. At least, not this time around. I did try to do this once upon a time using tun instead of tap, and it was mostly a crash and burn. But IIRC that was primarily due to the nature of tun.

The rest is just some standardish options.

So, I can hear you now, "you lunatic, look how much more work that was than just using the OpenVPN options". It may look that way, but in the long run it is not. I know what that up script is doing, at a glance. I know now. I know 3 months from now. I know a year from now. Reading OpenVPN options is like reading a second language. It takes effort and I just don't do it often enough, compared to the iproute2 version. I can remember what an up script does, and I can form a mental model of the network topology from that up script a lot easier than I can from the equivalent OpenVPN config, and that will save me time and frustration down the road, and it is definitely worth it.

As a bonus, I actually understand exactly where OpenVPN fits into the picture, and by putting it in its place I have expanded my understanding of the situation and opened the door to more advanced techniques down the road, should I find that I need them. Even if you take the road more traveled by, hopefully your mind will also have been expanded by reading this post. Happy VPNing!

Once upon a time, bash (or some other shell) introduced color in the shell prompt. The syntax was funky and all but illegible, so manpages and HOWTOs were written. The technical elite (i.e. high school kids and sysadmins with nothing better to do than configure their bash prompts) rejoiced.

Once upon a more recent time, I crafted this bash prompt:

PS1='$(r=$?; [ 0 == $r ] || echo "\[\e[33m\]$r\[\e[0m\] ")\u@\h:\w\n\[\e[32m\]\$\[\e[0m\] '

Actually, its history is more of an evolution than a crafting, but nonetheless I like it.

Then, out of the blue, someone broke bash. The \[ and \] non-printing character delimiters stopped working (or stopped working properly). Their purpose is to tell bash that the stuff inside is non-printing so don't count it when you're counting the number of characters in the prompt. This is vital for proper wrapping and tab completion in the presence of color escape sequences. I don't know who broke it or why, but removing \[ and \] seemed to fix the symptom.

Fast forward far too long, and someone fixed the problem. Once again you needed \[ and \].

Fast forward again, and it's broken. Or maybe we're just going around in circles on which version of bash is being used in various distributions and OS's. In this case, Ubuntu is still (or again) broken and OS X is broken in a different way. They're both running 3.2.x. The bug is probably the same and the manifestation is simply different.

The current manifestation of this bug is that neither with nor without the delimiters works. I'm forced to go back to a noncolored prompt, and I'm not happy about it. This bug has been around for at least a year. I've reported it several times to various distros. Why is it not yet fixed? If they fixed it today, we'd still be battling the bug for years to come because of the many distros and OS's that have assumed buggy versions of bash.

I was a beta tester for VMware Fusion. Fusion is a quality product, as I've come to expect from VMware. Unfortunately, the beta is over and a dialog box popped up when I tried to run it the other day. "The long wait is over!" Yeah, I've been waiting with bated breath for my beta to stop working until I fork over $40. You read my mind!

$60 ($40 after the current promotion) is the most affordable virtualization option for OS X right now, and in my opinion the most technically superior to boot. But I'm still a cheapskate, so I decided to use VMware Server on Linux over NX instead. NX is indistinguishable from magic. It works great, with one problem: the keyboard mapping in the virtual machine is completely skewampus.

In googling for the answer, I found a few people who had trouble with a key here or a key there. No, I had a completely unusable keyboard. e was the backspace key, backspace was a comma, escape was a letter, and everything else in between was equally unreasonable.

So I tried it over ssh (btw, it works a lot faster if you use ssh -Y instead of ssh -X), and it worked fine. So the gauntlet was down.

After some stabbing in the dark, reading, more stabbing in the dark, more reading, finally understanding, and one last stab in the dark, I got it to work. The theory is in this article. In short, VMware tries to map key codes to emulated PC scan codes (v-codes). If it can't do that, it maps keysym codes to v-codes. The former is apparently foolproof but isn't always an option. The problem here is that VMware thinks it will work, but it won't (probably because Apple's X11 isn't XFree86). So we simply need to put this in ~/.vmware/config:

xkeymap.nokeycodeMap = true

For the record, when you boot a system using LVM with a rescue CD you might have to do the following:

vgchange -a y $vgname

Where you get vgname via vgscan. Only then will you be able to mount the logical volumes (which you can find with lvscan), e.g.

mount /dev/VolGroup00/LogVol00 /mnt

This is one of those things that isn't too hard to learn, but when you need it is the last time you want to be learning it.

You know how clowns run around tripping over things and hitting eachother with mallets? That's what it feels like to be a pair of system administrators in the room when a production server is down in the middle of the day. Not the time to be wondering why you can't access your LVM root partition.

Back in the day, I cared about having my kernel as a Debian package. I used kernel-package, and I was happy. Then I started wanting to compile vanilla kernels and eventually the overhead just got to me and I stopped using it. I was not using kernel-package, and I was happy.

Recently, I revisited kernel-package for some reason I can't now remember. Wow, it has come a long way. Now it works with vanilla kernels, no fuss. Now it builds initrd images with a single command line option, no fuss. Now it is even easier than just building the kernel by hand (because I don't have to reconfigure the kernel from the stock kernel after make oldconfig due to the lack of initrd (no, I will never even try doing an initrd by hand)).

Here's all you have to do:

# download kernel source
cd linux
make oldconfig
# make menuconfig 
fakeroot make-kpkg --initrd kernel-image kernel-headers
sudo dpkg -i ../linux*deb<tab>

In case you haven't heard, Debian is planning on renaming Firefox to Iceweasel in its upcoming December release, codenamed Etch. The argument between Debian and Mozilla has been raging for quite some time now, and it appears they've reached a stalemate and so they will be doing the only thing they can do.

To recap, Debian won't distribute the Firefox and Thunderbird icons because they're copyrighted with a non-free license. Mozilla won't permit the use of its trademarked names if the official icons aren't used. Also, Mozilla wants complete control over the software, meaning Debian can't apply any patches of its own that haven't been approved by Ye Olde Mozilla. This is free software we're talking about right?

This phrase is repeated over and over when discussing the disagreement: "Mozilla understandably wants to maintain control and blah blah blah blah." Well I'm sorry, but while I can understand Mozilla's reasoning I can't agree with it and I do not have sympathy for them. Firefox and Thunderbird are big projects, but there have been a lot of bigger projects that have gone before who have not had issues with quality control and have not resorted to trademark bullying. Like, oh, the Linux kernel itself. Firefox will simply not ever have the variability that the Linux kernel has from distro to distro, no matter what distribution starts patching it. You don't see Linus throwing around his weight forcing people to distribute sanctioned vanilla kernels with Tux bootsplash images, or else.

Is Debian being anal? Well, yes. That's what Debian does, so people who need that assurance of utter freedom can sleep well at night. Debian has been around for a long time, and its guidelines haven't changed. They're like laws of nature. Mozilla is being a prick. Mozilla doesn't have to be a prick, but it certainly is being consistent with past behavior (I'm thinking of the Adobe SVG plugin scandal). It looks like Iceweasel is here to stay.

I'm divided on what happens next. I want Iceweasel to be released and be the name of the browser on Debian and Ubuntu for the next two years, because it will serve Mozilla right for being pricks. But, the responsible side of me says that can only be bad for everybody and hopes that Mozilla and Debian can come to some sort of agreement. Besides, I really like the Firefox and Thunderbird icons.

Read more at LWN

Announcing the latest version of hearnet: the (in)famous little JACK toy. The name may bring back memories of your favorite lunch lady, but the sound of your own network traffic will move you in ways that she never could have hoped to move you. I hope that's a good thing.

This release sports filter expressions (a la tcpdump) and the ability to drop privileges and run as a normal user, and therefore play nice with the other JACK apps in your playground.

Enjoy, and don't get any hair in your network.