The Fugue

I love Ruby, and I use it whenever I can for all kinds of tasks. When I was working as a professional system administrator, I put it to good use a few times there too. But using ruby always left me feeling a little guilty, because whoever came after me would probably curse my name for using a "nonstandard" language. It didn't help that Ruby was rarely installed, if even available, by the popular server distributions of the day. But all that has changed in the past two years, and using Ruby is often acceptable even in the conservative world of system administration.

You can imagine my interest then, in a book titled "Practical Ruby for System Administration" by André Ben Hamou. The title is telling; Ruby is known for being recommended by The Pragmatic Programmers because, well, it's a pragmatic language. System administration is all about pragmatism, above all else. Doing things right is important, but doing them now is more important. I knew all along that Ruby was an excellent tool for the enlightened sysadmin; now there's a book to back me up.

The book starts out with why you would want to use Ruby, and nips the counterarguments in the bud. Hamou does a good job on both counts, though as I already know Ruby I can't say whether the language introduction is sufficient of its own accord to get started with actually writing Ruby, or whether you'll need to refer to other sources (which are available online).

In chapter 2 he discusses one-liners. One-liners are the staple of many sysadmins, but I never fell for them. I'd rather write a throwaway script than try to cram it all on one line, or if it can really be done well on one line it can probably be done even more concisely with shell script and UNIX tools. However, the chapter does discuss a lot of fascinating switches that the ruby executable takes. I can see myself using many of these, and I was ignorant of most of them. You might learn the same from a careful study of the man page, but Hamou presents it in an easier-to-digest format. It's not always easy to inject humor into a book while maintaining concise brevity, but Hamou does a decent job of that. I found myself laughing aloud more than once, and yet I feel that the book is concise enough to serve as a reference, and that the jokes won't get annoying after a few readings of a section.

This book is not only a great book for the sysadmin hoping to use Ruby, but also an excellent book for any sysadmin who may not even be interested in Ruby. Although not even pretending to be a book on system administration best practices, there are many gems in here that will leave you saying "Why didn't I think of that?" and "I really need to implement that. It will be so helpful and it's astoundingly simple to do with Ruby!"

Chapter 3 gives you not only a quantitative feel for the speed of Ruby versus other performance-oriented languages (i.e. C), but it teaches you when to be concerned with execution speed, when to be concerned with implementation speed, and makes you think twice about your own feelings about performance. "Ruby is slow" is one of the favorite counterarguments against Ruby, but it's rarely a good one. This is even more true in system administration, where the sysadmin's time is much more important than whether the script runs in 1/10 second or 3 seconds.

Chapter 4 discusses "metaprogramming". I found myself at odds with Hamou most in this chapter, but it was all academic differences in terminology. He's fairly sloppy with the term metaprogramming and other terms (e.g. "macros") in this chapter, but the content is nonetheless a useful and vital part of making the most out of Ruby. I even learned a thing or two. He discusses domain-specific languages (DSLs) in Ruby, but mostly at the level of recognizing one when you see one. I think the world needs a paper or book on rapidly making your own DSLs, something sysadmins could really leverage if it were truly easy to do. I've written a DSL in Ruby, and while easier than I could possibly have imagined it's still not at the level of "easy and completely generic" that I feel DSLs can one day reach.

Chapter 5 is where the fun really begins. With the basics of the language under our belts we can really look at specific examples. This is where the book really shines, both as a tool for applying Ruby and as a minefield of good sysadmin practice. We learn to read and write files, with examples for the most common ones. I don't mean we discuss IO.open and IO.close in detail, I mean we talk about the concepts that apply across all kinds of file reading/parsing and generation, including issues of locking.

In chapter 6 we explore the storage and retreival of data, in a variety of approaches including inspect, marshalling, YAML, and ActiveRecord. Most interesting to me was the section on memcached. I think he ought to also have introduced SQLite (perhaps in the context of ActiveRecord) and DRb.

Chapter 7 is dedicated to dealing with "enterprise data". You know that of which we speak. XML, CSVs, protocols like XML-RPC, SOAP. Most valuable to me in this chapter was a coherent discussion of what REST is (finally!) and how to do it in Ruby.

Chapter 8 discusses network, including writing simple clients and servers. One tendency Hamou has in this book is to use pure Ruby and steer clear of system and backticks. This tendency sticks out in this chapter, where much time is spent discussing that which could be acheived better with a shell script, or at least a Ruby script with judicious use of system or backticks. The general argument for doing things in pure Ruby is portability, and to a lesser extent performance. Neither of these is the first concern of a sysadmin, who is generally not going to write a script that must work on more than one platform (even if it runs on several different Linux/UNIX distributions, which have the same GNU tools).

Chapter 9 deals with that task which we all hate: network and log monitoring. He has some gems of wisdom herein, but all in all we're left feeling about the same as when we went in: we can do it (now we can do it with Ruby) but it's a pain. The gains are not as great here as they are in other areas. This isn't Ruby's fault, nor Hamou's, it's just that nobody's really come up with a good and general way to attack this problem yet.

Chapter 10 discusses RubyGems. I think it's fitting that such a chapter be at the end of the book, and that's all I have to say about that today.

Chapter 11 discusses testing. You should do it in some form and he discusses a few forms. Nothing earth-shattering here, from where I sit. Chapter 12 talks about the future of Ruby, and will probably cause you to salivate.

All in all, an excellent book on using Ruby as a sysadmin. Go ahead, come out of the closet. Ruby is not only OK, it's often the best choice.

Preface: this is a long rant. If you just want your joystick hat to work in X-Plane, download jhat and follow the instructions in the README.

Linux reports joystick hats as axes, because that's what the device reports them as. Windows drivers often put up a façade where the directional hat is translated into 4 buttons. X-Plane is Windows-centric and therefore directional hats don't work in X-Plane in Linux.

I submitted a bug report, and was told

i cant do that
i have to use what is reported
but you can assign an AXIS to look around.. it is one of the joystick axis options...

Actually, you can't use the "look around" axis, because the joystick hat is not analog, but tristate. It's either left, middle, or right (or up, middle, or down for the vertical axis). So you are either looking behind you, in front of you, or behind you - not the most useful combination. I informed him of the situation, and provided him with some pseudocode for handling the axes. I'm always amused when developers try to delude the user and themselves into thinking some bug or feature is impossible to fix or implement. Human nature is a funny thing. Sometimes we even believe it ourselves. His reply to my information and pseudocode:

the code would be easy, but i wont write a whole new type of axis response just for a few computers that are not working right
it is too much confusion for too little gain
sorry!

Wow, this task has gone from being impossible to being easy in one short day! Well of course it's easy. I knew that. He knew that. Now we're on the same page. The real reason he won't do it is because he doesn't want to. It would be easy, but it would be inconvenient. And those darned pesky Linux freaks aren't worth the trouble. Now, at this point I'm struggling to be cordial, but I believe it's important to remain cordial until the bitter end. I replied with my condolences that he has so few Linux customers that he shouldn't care if his software is broken in Linux. I also tossed out some more ideas - a different type of axis behavior for scrolling instead of absolute positioning, etc. His reply,

i have very few linux users, this is true

also, it seems obvious to me that the OS should report the hardware properly... not that the app should work backwards and add more confusion for EVERYONE because ONE person did not do his job properly!

Now he's getting defensive. The truth hurts sometimes. Let's step back and review. The joystick reports the directional hat as 2 axes. It's a HID device, so there is no need for special drivers. However, somewhere along the line (I don't know the history), some joystick driver writer in Windows (probably before HID even existed) decided that directional hats should be represented at 4 buttons. Does one make more sense than the other? I don't think so. I think they're both valid viewpoints. I'm not surprised the engineers chose to use axes. I'm not surprised the programmers wanted to use 4 buttons. Two valid viewpoints. What we have, however, is a lack of standards. There is nothing to my knowledge that says joystick hats should be buttons, it's just Windows convention. Windows convention is fine and dandy in a Windows world, but Windows convention doesn't stand up very well as a challenger to the appropriateness of faithfully representing what the HID device itself reports. This is not about a Linux developer slacking on the job. This is about the world being a complicated place where sometimes things are implemented differently for perfectly valid reasons. If you want to play the portability game, you have to deal with this stuff. He knows that, I'm sure. It's just laziness, hubris, and now a personal resolution to not give in to this pesky Linux freak.

One more cordial reply from me, stating that both can be right, and a terse reply from him:

no, i think linux is wrong here
a button is a button, not an axis

I didn't reply anymore, it would serve no purpose. I could say, "an axis is an axis, not a button", because it is an axis after all. But I see I'm not going to change his mind or convince him. So I took a different tack. I visited the old forum thread with the not-really-feasible joydev.c patch that others came up with and encouraged them to encourage the author to consider treating us Linux customers to a simple bug fix.

But the gauntlet was down. I can't just leave a technical problem like this, that should be so easy, in eternal limbo. I have to see if it can be solved. I read through joydev.c in the kernel and thought about what it would take to make a robust patch that wouldn't have problems with plugging things in in different orders and wouldn't ruin other joysticks' behavior. I thought of a way, but I didn't implement it. I decided there had to be a better way. An easier way. A userspace way. And so there was.

I knew it was easy to read the joystick events in user space. I figured there just had to be a way to synthesize the fake joystick events that I needed. I tried to grok the input core and event subsystems. I read articles in Linux Journal. I groped through kernel changelogs and looked high and low for what this weird uevent file in sysfs was (completely unrelated—it's a buffer to aid hotplug in managing coldplug events). I exhausted poor Dr. Google. And then I stumbled upon a manpage for the XTest X extension. In a flood the blinds were lifted and I realized that what I needed was not to synthesize joystick events (hard), but to synthesize the keyboard events (easier). Not only that, but I didn't have to synthesize them at the kernel level, I could synthesize them in X (easy).

An hour or two later I had finished the task. It's a beautiful thing, to be able to look around in X-Plane. I call it jhat because I was feeling pretty unimaginative at the time. It uses xautomation to do the XTest stuff because I was feeling pretty lazy at the time. Grab the tarball.Now I have two flight simulators that I can stand to fly. I prefer FlightGear for many reasons, not the least of which is this whole fiasco (I could have submitted a patch in the first place and nobody would have gotten hurt). But I do like X-Plane because some of the planes that are available I do so enjoy flying, like the Mooney M20K.

So, as a follow up to my earlier post on respecting your bug reporters, let this be a lesson to you developers out there: every user matters. Especially paying users. If you don't want to support Linux, then by all means don't support Linux. You're not fooling anyone when you do a half-baked job of supporting his platform.

Back in the day, I cared about having my kernel as a Debian package. I used kernel-package, and I was happy. Then I started wanting to compile vanilla kernels and eventually the overhead just got to me and I stopped using it. I was not using kernel-package, and I was happy.

Recently, I revisited kernel-package for some reason I can't now remember. Wow, it has come a long way. Now it works with vanilla kernels, no fuss. Now it builds initrd images with a single command line option, no fuss. Now it is even easier than just building the kernel by hand (because I don't have to reconfigure the kernel from the stock kernel after make oldconfig due to the lack of initrd (no, I will never even try doing an initrd by hand)).

Here's all you have to do:

# download kernel source
cd linux
make oldconfig
# make menuconfig 
fakeroot make-kpkg --initrd kernel-image kernel-headers
sudo dpkg -i ../linux*deb<tab>

This started out as a comment on Redbeard's Musings on Ubuntu, but grew into something worthy of being an article in itself. Enjoy!

As a Debian enthusiast and package maintainer, I feel qualified to comment. I am not one of the Debian elite, you'd have to pay me (or at least reimburse me) to go to a Debian conference, and I don't have the title "Debian Developer" which means not only do you maintain packages but you jump through hoops too.

Debian has a very rigid policy. All package maintainers have to maintain their packages in accordance with this policy or face the wrath of their peers. It is widely believed (and I share this belief) that the high quality of the Debian distribution is directly proportional to the adherence of packages to policy.

Now all of a sudden here's this upstart Ubuntu that builds on Debian. That's fine, and while there might be the occasional DD that feels jealousy that Ubuntu is building on Debian and getting all the credit, if you look around you realize there's dozens of projects built on Debian. It's a common thing, and encouraged. There's even infrastructure and packages in place to make it easier to roll your own Debian-based distro.

The irritation to the Debianistas is that Ubuntu doesn't always follow policy. They cut corners. They package nonfree software. They do all sorts of things that break policy. This is on the one hand annoying to the debian maintainers because they can't cut corners, but also because they can't just import the changes wholesale back into debian. They have to modify ubuntu's changes to fit policy. It's like someone not-quite-forking your project and sending you a deluge of patches that don't fit your coding style. You're thankful and you're annoyed. It's a love-hate thing. In some cases it causes real problems, not just annoyances.

So the Debianistas who think Debian is all that and a bag of chips naturally get annoyed when Ubuntu comes and does all this, which of course it does in the name of THE USERS and all that is high and mighty, but the Debianistas are perfectly happy with the state of usability in Debian, it rubs the wrong way. Ubuntistus gets mad because Debianistas are heartless elitist pigs, Debianistas get mad because Ubuntistus are busybody carelesss l33t hax0rs, secretly allied with the evil Gnome conspiracy to take away our user interface freedom.

It's not a matter of either Debianistas or Ubuntistus being right or wrong, or better or worse. It's just that common thing called differing viewpoints and priorities in close contact. The people wearing those shirts are immature socially-inept geeks. What's new here?

For the record, I like Ubuntu and although it has annoyed me on occasion (it annoyed me as a Debian user because it doesn't follow The Debian Way all the time), it has also made sharing Linux with others much easier and more worthwhile, and it has given Debian a kick in the pants which can only help. And I love the color scheme. Here's to a continuing symbiotic (if not entirely friendly) relationship between Debian and Ubuntu.

I have a Yepp, which is a wonderful mp3 player because it's small (the size of a AA battery), plays OGG (out of the box!), and presents itself as USB mass storage.

As I am frequently changing the contents for my yepp, because I use it mostly for listening to podcasts, I decided the best thing to do is to keep a mirror on my hard drive and then just hook up the yepp and sync it. Because I also keep music on for relatively long time (compared to podcasts) and I let my podcasting software handle deletions so some longer-period podcasts hang around for awhile, I figured something like rsync or unison would be the best approach.

Initially I was excited about unison because I expected I would sometimes change stuff on the yepp and want to see the changes propogated to my hard drive. Possible scenarios for this are putting random files or songs on the yepp from other computers or deleting podcasts I had already listened to on the yepp.

I quickly found that unison was a very poor choice. Unison is designed for synchronizing incremental changes to roughly the same set of files. As such, it expends a lot of effort in doing things like checksums and who knows what else. The first time you run unison on a set of files it will take a long time, this is a FAQ. And by long time I mean long time. I once timed how long it takes to copy over a bunch of files that filled all 512MB. It took about 5 minutes IIRC. I then ran unison on the same set of files and after about 5 minutes it had completed almost nothing and had an estimated completion time of an hour. I am not privy to the internal workings of unison, but I think the constant addition of new files, which is in fact basically the only change, as the MP3s themselves don't change, meant unison had to do whatever this really slow thing is fresh for every new file. I have found that it doesn't matter if unison has already built its database or not, adding new files always takes 10 times or more longer than it would to just copy them. I'm not sure what it's doing, but I do know that the whole time my yepp is flashing between "READY" and "WRITING". That tells me unison is doing a lot of itty bitty writes to the flash memory, and I don't think that can be good (yes, even when you specify rsync transfer mode).

So I gave up on the bidirectional sync and wrote a simple script to sync with rsync. The first version looked like this:

rsync --delete -rv ~/Music/Podcasts /Volumes/YEPP

This syncs my Podcast folder to the Podcast folder on the yepp. It takes no longer than cp to copy new files, and it deletes old podcasts. Which reminds me, unison waits until the end to delete old files. This is not good if you're tight on space. rsync deletes the old files first, then copies the new/changed ones over.

There's a problem with that script, though. When you run it again tomorrow, it will think it needs to sync all the files. I think that rsync will not actually copy the whole file over, but it will have to do the checksum thing which is slow, and you'll get basically no speedup. And my yepp keeps flashing writing even when no changes would need to be propogating, so rsync is writing to my flash more often than it needs to too.

Well the problem isn't rsync, it's the user. With only the -rv options, (recursive and verbose) rsync will copy new files over with the current timestamp. When you run it again, rsync's quick check algorithm checks the timestamp and sees that they're different and assumes it has to sync those files. It then does the checksum calculations and only the changed bits are transferred over the network. Good for the network usage, not too bad if you have smallish files and a fast disk. Not very good at all if one of the disks is slow.

So we need to be sure and sync the modified times too. That option is -t, which is included in -a. So I tried this script:

rsync --delete -av ~/Music/Podcasts /Volumes/YEPP

That kind of sort of worked. Some files weren't resynced, but others were. I tried -t and a few other things, and scratched my head a bit. Then I read the fine manual (rsync(1)).

When transferring to FAT filesystems rsync may re-sync unmodified files. See the comments on the --modify-window option.

Here's what it has to say about --modify-window:

When comparing two timestamps rsync treats the timestamps as being equal if they are within the value of modify_window. This is normally zero, but you may find it useful to set this to a larger value in some situations. In particular, when transferring to Windows FAT filesystems which cannot represent times with a 1 second resolution --modify-window=1 is useful.

So I updated my script and now it works like a charm:

rsync --delete --modify-window=1 -av ~/Music/Podcasts /Volumes/YEPP

There are a few lessons to be learned here. First, unison isn't always the best tool for the job. Don't drink the kool-aid. I think it is a good bit of software though and has its uses. I'm not sure it can be adapted for efficient flash-drive frequent new-file usage, but if you know how feel free to comment. Second, rsync is nifty, but it's more nifty when you use it properly. Get in the habit of -a instead of -r, it almost never hurts and will often speed things up dramatically. Remember the --modify-window=1 trick for FAT filesystems. Finally, if you *do* make a change to a file and keep the same timestamp and file size somehow, rsync won't sync that file unless you specify -c. It's a pathological case, but it's good to know about.

If when you upgrade MPlayer the wonderful VIDIX output mysteriously stops working (I have a Radeon card, but I don't think it matters), Don't Panic.

It's probably just that you forgot to chmod +s /usr/bin/mplayer (or run mplayer as root, you decide which is the lesser evil). Incidentally, if you know the device file to change permissions on so that you can run mplayer as user without the setuid bit, I'm theoretically interested in acquiring that bit of knowledge.

Scenario: you have a bunch of avi files (or whatever) that you want to make into a DVD. They may be episodes of your favorite comedy that you recorded with MythTV and want to keep around but not on your hard disk. They may be home videos. Whatever.

There are commercial programs to do this, and they are either expensive or inflexible and for any of a million reasons they don't meet your need. What do you do? There's a lot of good information about the details out there, with the exception of how to transcode your video, so I will give the overview, some links, and details on how to transcode.

First, what a DVD is. It's an ISO9660 filesystem following a certain convention. That convention is that stuff has certain names in a certain filesystem layout, and that stuff is encoded in certain ways. But it's just a regular filesystem underneath. If you look at a DVD in the shell you can see what I mean.

Assuming you want to play video DVD on your Wal-Mart-purchased consumer electronics, you need to get your video into an MPEG-2 video with AC3 or PCM audio at 48khz, with the appropriate frame rate for your TV and the correct size and aspect ratio. The mencoder docs have a nifty table showing what you need to decide. Once you've decided, use ffmpeg to transcode. It's not hard: ffmpeg -i in.avi -target ntsc-dvd -acodec ac3 out.mpg. You can change the bitrate with the -b option (see the myriad bitrate calculators online if you're trying to squeeze things. I've found the bitrate calculator in ffmpegX to be the best to use, if you have OS X). You can change the audio bitrate, e.g. -ab 192. You can change the size from "full DVD" to "half DVD" with -s 352x480. You could play with lots of options. What's important is that you see something like this:

Output #0, dvd, to 'out.mpg':
Stream #0.0: Video: mpeg2video, yuv420p, 352x480, q=2-31, 1703 kb/s, 29.97 fps(c)
Stream #0.1: Audio: ac3, 48000 Hz, stereo, 256 kb/s

If it doesn't look right, it's time to double check the options and the order of the options. That one bit me, but it's right there in the manpage:

ffmpeg [[infile options][-i infile]]... {[outfile options] outfile}...

If it looks right, but mplayer doesn't seem to grok the output, then get a new ffmpeg. I had one (ffmpeg from darwinports) that would seem to do everything right but somehow leave out or corrupt the audio, so my transcoded videos had no audio. So I compiled my own from SVN.

Once you have transcoded the videos, it's time to use dvdauthor to make menus and create the DVD filesystem. It'll take a little work and XMLing, but it's well-documented online. Then you need to burn your image to a DVD.

But before you burn, try it out in-place with your favorite DVD watching program. Pointing it at the VIDEO_TS directory or perhaps VIDEO_TS/.. should do the trick. If it loads and plays, you're probably good to go. Burn and test. Get out that sharpie and decorate. Enjoy your newfound sense of accomplishment!

I just finished reading this article on Upstart, a replacement for init.

I've long been dissatisfied with the System V init scheme. I've looked at file-rc, runit (which is similar to daemontools), and briefly at some others that have popped up from time to time (I think I remember one that had to do with purple cows, but my memory is hazy). None of them has really done anything for me. file-rc is a nice concept taken not quite far enough (it's almost as much a pain editing that file format as dealing with symlinks), and it's not an init replacement in any case. runit/daemontools are very handy, and I use them, but when I tried to replace init with runit I made short work of hosing my startup system. As easy as it is to write run scripts, I do not want to write one for each daemon that I do now or ever will install.

But upstart is very promising. Its coolness factor is higher than any of the others, and it has an upgrade path. In fact, Ubuntu will be using upstart in the upcoming release of Edgy Eft, and is already in use in Ubuntu testing, completely replacing sysvinit. Kudos to the Ubuntu folks for recognizing the need for something better and implementing it. I hope that replacing sysvinit with upstart (in an apt-get sort of way) in Debian becomes a reality soon. In any case, I'm going to give it a test on my laptop when I get some spare time.

Or more specifically, huzzah for the Debian developers involved in finally doing what I would have done years ago in their place: forking cdrtools and showing Jörg Schilling the door.

Jörg has caused me endless troubles because of his antics. My burner works, then doesn't work, then works partially, then doesn't work at all. The only change is the kernel version or the cdrtools version, and the only reason is that Jörg is on some loony ego-stroking crusade. Good riddance.

Thank you Debian, and thank you Sun for giving Jörg the rope to hang himself with.

It seems I was a little premature with my last post. It does work just as I said on Debian, but the scanner wouldn't work on Dapper.

First, the problem in detail. There's actually a couple problems. The first is that the sane backend Samsung provides apparently has a bug that prevents it from working remotely. That is, I can't scan on gwythaint (my Dapper box) when the scanner is attached to falcon (my Debian box). Yes, sane can do this. The client crashes with a complaint about memory allocation. This is why I decided to move the printer over to the Dapper box. This problem can only be fixed by Samsung, and hopefully they will do so but I'm looking for a now solution.

On Dapper, the scanner was not found at all, but the printer worked fine. A little detective work revealed that Samsung's sane backend is looking for /dev/usb/lp0, which does not exist. You need to symlink /dev/usblp0 to it, which you can do with this entry in /etc/udev/rules.d/10-local.rules (create it if necessary):

BUS=="usb", KERNEL=="lp[0-9]*", SYMLINK+="usb/%k"

That didn't do the whole trick, though. The version of libusb on Dapper looks in /proc/bus/usb which is not mounted by default on Dapper. (At least, it's not mounted on my Dapper which is running a vanilla 2.6.17 kernel) It seems the latest and greatest way to mess up udev/usb (I swear they sit up late at night thinking of how to mess this up) is to move /proc/bus/usb to /dev/bus/usb. The fix is easy. Add this line to /etc/fstab:

none            /proc/bus/usb   usbfs        defaults        0       0

Then mount /proc/bus/usb as root.

I still haven't figured out why, but no matter the permissions on any of the above, any libsane-using program will segfault if it is not setuid root (or run as root). If anyone can figure that one out I'd love to hear it, but I probably won't fuss with it much more myself. See the output of strace scanimage -L when not root.