The Fugue

Stuart and I came up with a refinement to my favorite bash idiom today. It was:

find ... | while read FOO; do ...; done

as demonstrated in:

find . -name '*.mp3' -type f | while read MP3; do
    ffmpeg -i $MP3 ${MP3%.mp3}.m4a
done

This handles whitespace (except newlines) in filenames. Except that ffmpeg is sharing stdin with while, which can cause some very funky problems. This is safer:

find ... | while read FOO; do echo | (
    ...
); done

and while you’re at it, you might as well take into account the unlikely-but-possible corner case of a filename containing a newline:

find ... -print0 | while read -d $'\0' FOO; do echo | (
    ...
); done

Not literally, of course. This is programming talk, those of you who aren’t programmers can let your eyes glaze over.

I wanted a script to start a bunch of little servers, then wait around for them to finish or when the user interrupts with Ctrl-C, clean up the servers instead of orphaning them. I wanted to propagate the SIGINT to the child processes. I wanted to kill the kids.

The simple way, if you just want to make sure the kids are killed and you don’t care how:

sleep 300 &
# etc.
trap "kill $(echo $(jobs -p)) 2>/dev/null" EXIT
wait

If you only want to trap SIGINT and want to make sure you send SIGINT (not SIGKILL) to the children, then you want to do something like:

trap "kill -INT $(echo $(jobs -p)) 2>/dev/null" INT
wait

Update: I was asked by a shell scripting guru why I needed to do $(echo $(jobs -p)) and not just $(jobs -p). I intended to cover that but forgot. The reason is that $(jobs -p) has newlines and while that’s not usually a problem it is in a trap statement, because it’s evaluated at creation time not at run time. It also means that processes created after you create the trap wouldn’t be killed. Then, he suggested a function instead. Pure brilliance. Where does he come up with these things? Here’s the improved version:

function killkids() { kill $(jobs -p); }
trap "killkids" EXIT

You can still redirect stderr if you want to, but the reason I was directing stderr was because some of the kids may have already died (early evaluation remember) and then kill would needlessly complain. This way, it kills all the kids that are still alive, none more none less.

“What is my IP?” A frequent question while we all remain under the oppression of NAT. Of course most of you are familiar with whatismyip.com and friends, but did you know you can do the same thing yourself very easily? All you need is a webserver (across the NAT in question, of course).

Here’s a CGI version:

#!/bin/sh
echo "Content-type: text/plain"
echo
echo $REMOTE_ADDR

If CGI is a pain but you have PHP:

<?php
  header("Content-type: text/plain");
  echo $_SERVER['REMOTE_ADDR'];
?>

Both of these are suitable for scripting, e.g.

#!/bin/bash
URL=http://fugal.net/ip.cgi
echo Your IP address is `curl -s "$URL"`

I’ve been exploring D the language. I really do like it, but that’s another post. There are a couple of D compilers, but the only viable option on OS X seems to be gdc. I installed it via MacPorts. On Leopard, gdc generates assembly that makes the FSF gcc complain “indirect jmp without `*'” over and over. The bug is known, and other than being annoying it seems harmless.

So I decided what I needed was a script that would filter out these frivolous warnings without otherwise affecting stderr, and also without changing the exit status (so make can do the right thing). This turned out to be easier said than done. Finally I stumbled on the right incantation:

#! /bin/bash
gdc=/opt/local/bin/gdc
msg="indirect jmp without"
$gdc "$@" 2> >(grep -v "$msg" 1>&2)

We redirect stderr to the named pipe corresponding to that subshell (see “Process Substitution” in the bash manpage), then we redirect grep’s output to its stderr. Because grep is in a subshell, its exit status doesn’t mess up the exit status of the script, which is the exit status of gdc, as it should be.

Have you ever been frustrated with perfectly valid bash syntax being highlighted as incorrect in vim? Like this:

#!/bin/sh
foo=$(ls /tmp)

$() is a perfectly valid, nay preferred substitute for backticks. The problem is that vim is deciding this is pure old bourne shell instead of whatever else we’d like it to be. If you change the shebang to #!/bin/bash and re-edit the file, then the error markings go away. But maybe you’re writing for the nebulous POSIX shell, not bash nor sh. Or maybe you just don’t care and you don’t want vim complaining that you’re using bashisms even though your shebang says sh. You can set the defaults so that it reflects your system and preferences. It’s all there in :help sh.vim.

In the spirit of release early, rewrite often, I have released Crème Rappel version 2. Version 1 was a shell script that combined Growl and at. Then Apple released 10.5.2 not half a week later and broke at altogether. Sick of fighting with launchd and other Apple superiority complexes, I set about to nurture my own superiority complex and rewrite Crème Rappel to be completely independent of at.

Of course, that’s getting too heavy for a shell script, so I moved to Ruby. One thing I didn’t want was to require a daemon to be running. Daemons can fail or
forget to start up, and that means I couldn’t really truly trust the tool. The recent at debacle is just another case in point. So, instead I wrote Crème to fork a process that sleeps until the moment of truth, then fires off the reminder. It turns out the obvious function for the job, sleep(), is a poor choice here. In fact, every timer I tried had the same problem, including one I thought would not: setitimer(). When you suspend the laptop, it appears you also suspend time. If you don’t believe me, try this simple experiment:

date; sleep 30; date

Put the laptop to sleep during the sleep for a substantial time, then notice that when you resume you still have to wait for the full 30 seconds to tick by
even though it has actually been a minute plus since you issued the command. So I sleep for one second intervals instead, checking the time every time.

This is not just a backpedaling rewrite, though. It also adds more flexible and easy-to-type timespecs, and a spiffy website.
If you give it a try and it doesn’t work, or you struggle with the documentation, please do drop me a line so I can fix it. I want it to be worth
every bit of bandwidth that you paid for it.

I discovered a very strange bug today with OS X’s Bourne shell. If you have OS X, give this a try:

/bin/sh -c 'echo -n bug'

This is what you should see:

$ /bin/sh -c 'echo -n bug'
bug$

This is what I see:

$ /bin/sh -c 'echo -n bug'
-n bug
$

In other words, it’s ignoring the -n option. It works fine in bash, it’s only sh that’s broken. It gets better though. If you’re using iTerm instead of Terminal.app, it works fine. I have combed through the environment, the locale settings, the terminal emulation, and I can’t account for it. I’ve tried ssh from a linux box which behaves the same as Terminal.app (broken). Who knows what black magic iTerm is invoking.

So I replaced my bash and sh with the ones from MacPorts:

sudo port install bash
sudo mv /bin/bash /bin/bash.old
sudo mv /bin/sh /bin/sh.old
sudo ln /opt/local/bin/bash /bin/bash
sudo ln /opt/local/bin/bash /bin/sh

While you’re at it, feel free to

sudo port install coreutils +default_names

Problem solved. Very odd, though. abcde uses echo -n heavily, which breaks in all sorts of ugly ways before this fix. This patch causes abcde to use /bin/bash instead of /bin/sh:

Index: abcde-2.3.3/abcde
===================================================================
--- abcde-2.3.3.orig/abcde  2007-12-07 18:46:36.000000000 -0700
+++ abcde-2.3.3/abcde       2007-12-07 20:57:17.000000000 -0700
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
# Copyright (c) 1998-2001 Robert Woodcock <rcw@debian.org>
# Copyright (c) 2003-2005 Jesus Climent <jesus.climent@hispalinux.es>
# This code is hereby licensed for public consumption under either the
Index: abcde-2.3.3/cddb-tool
===================================================================
--- abcde-2.3.3.orig/cddb-tool      2007-12-07 20:56:49.000000000 -0700
+++ abcde-2.3.3/cddb-tool   2007-12-07 20:57:19.000000000 -0700
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash

# Copyright (C) 1999 Nathaniel Smith <njs@uclink4.berkeley.edu>
# Copyright (C) 1999, 2000, 2001 Robert Woodcock <rcw@debian.org>

Speaking of abcde, here’s a patch to get rid of an unrelated bug in 2.3.3:

Index: abcde-2.3.3/abcde
===================================================================
--- abcde-2.3.3.orig/abcde  2005-08-25 16:43:27.000000000 -0600
+++ abcde-2.3.3/abcde   2007-12-07 18:46:36.000000000 -0700
@@ -1946,7 +1946,7 @@
                        FILEPATH=$(find "$FILEPATH" | grep "/$REALTRACKNUM ");
                        # If the file exists, copy it
                        if [ -e "$FILEPATH" ] ; then
-               nice $READNICE $CDROMREADER "$FILEPATH" "$FILEARG" $REDIR
+               nice $READNICE $CDROMREADER "$FILEPATH" "$FILEARG"
                        else
                                false
                        fi ;;

Finally I have abcde at my fingertips again. There’s no replacement for abcde when it comes to ripping CDs.

Once upon a time, bash (or some other shell) introduced color in the shell prompt. The syntax was funky and all but illegible, so manpages and HOWTOs were written. The technical elite (i.e. high school kids and sysadmins with nothing better to do than configure their bash prompts) rejoiced.

Once upon a more recent time, I crafted this bash prompt:

PS1='$(r=$?; [ 0 == $r ] || echo "\[\e[33m\]$r\[\e[0m\] ")\u@\h:\w\n\[\e[32m\]\$\[\e[0m\] '

Actually, its history is more of an evolution than a crafting, but nonetheless
I like it.

Then, out of the blue, someone broke bash. The \[ and \] non-printing character delimiters stopped working (or stopped working properly). Their purpose is to tell bash that the stuff inside is non-printing so don’t count it when you’re counting the number of characters in the prompt. This is vital for proper wrapping and tab completion in the presence of color escape sequences. I don’t know who broke it or why, but removing \[ and \] seemed to fix the symptom.

Fast forward far too long, and someone fixed the problem. Once again you needed \[ and \].

Fast forward again, and it’s broken. Or maybe we’re just going around in circles on which version of bash is being used in various distributions and OS’s. In this case, Ubuntu is still (or again) broken and OS X is broken in a different way. They’re both running 3.2.x. The bug is probably the same and the manifestation is simply different.

The current manifestation of this bug is that neither with nor without the delimiters works. I’m forced to go back to a noncolored prompt, and I’m not happy about it. This bug has been around for at least a year. I’ve reported it several times to various distros. Why is it not yet fixed? If they fixed it today, we’d still be battling the bug for years to come because of the many distros and OS’s that have assumed buggy versions of bash.

This is the 21st Century. It’s time for Unicode. OS X does a decent job of supporting Unicode overall, but it is far from perfect. Perhaps the most noticeable is the mass confusion that is Unicode in the terminal. The problem is actually legion, and it takes a little effort to fix it. But you can fix it and you can do it today.

Start at this article on Internationalizing the Mac OS X terminal. Following the instructions there will get you 90% of the way there. You will install a new version of bash and coreutils (for ls, cd, etc.), you’ll set up your locale information (actually you only need to set LANG, in my experience. I export LANG=en_US.UTF-8 in my .bashrc).

If you use OS X’s default vim (/usr/bin/vim), it will Just Work. But if you have vim 7.0 from some other source, it may or may not work. In particular if you do sudo port install vim you’ll probably get a broken vim. Be sure to use the multibyte variant if you install from ports.

If you use iTerm, it might work worse than before you made the changes above. But notice that it works fine if you run another bash instance, or screen. I’m not sure but I think you fix it with the LC_CTYPE hack detailed in this article. Those .inputrc lines probably won’t hurt either, though I’m not sure what problem they fix. One of those two changes seems to have fixed the iTerm problem.

Speaking of screen, you probably want to add defutf8 on to your ~/.screenrc.

I’m curious if things work out of the box in $YOURFAVORITEDISTRO. I’ve previously mucked about with unicode so much in Debian and Ubuntu I have no recollection of what the default state of affairs is. Try the following and let me know in the comments how it worked.

echo æøÜÉ > /tmp/ƒøø
ls /tmp/ƒøø
cat /tmp/ƒ<tab>
vim /tmp/ƒøø
screen vim /tmp/ƒøø

For me, I see exactly that and exactly what you’d expect on the terminal and in vim (both in and out of screen), which has never all happened in OS X for me before. It does work over ssh on my Debian box, but as I say I’ve played with things there so it may or may not have worked out of the box.

Once upon a time I fought the good battle trying to get one of [gkwx]dm to run
a login shell, because you know, we’re logging in. I wanted it to run a
login shell so I could run keychain
or something.

Well today I wanted to figure out how to keep startx from running keychain.
It shouldn’t have been running keychain because keychain runs from
.bash_profile which shouldn’t be sourced for two good reasons: A) it’s not a
login shell (this is a boot script to start freevo), and B) it’s not bash,
it’s /bin/sh (yes, I know that’s really bash but when invoked as sh it doesn’t
source .bash_profile).

It turns out some bright kid created /etc/X11/Xsession.d/91source_profile (I
think that’s what it was called), that sourced every profile file it could
think of whenever Xsession was run. This is both bad and stupid. It’s bad
because sometimes an X session is not a login. It’s stupid because they could
have achieved the same thing in a more elegant way by adding --login to the
shebang line of /etc/X11/Xsession. The place to do login shell stuff is where
you actually log in, i.e. [xgkw]dm, not Xsession.

Please, my friends, if you go out into the world and work on something that
involves login and environment, read the manpages and understand what a login
shell, an interactive shell, and a noninteractive nonlogin shell are and when
you want each. Thanks. End of rant.