The Fugue

Stuart and I came up with a refinement to my favorite bash idiom today. It was:

find ... | while read FOO; do ...; done

as demonstrated in:

find . -name '*.mp3' -type f | while read MP3; do
    ffmpeg -i $MP3 ${MP3%.mp3}.m4a
done

This handles whitespace (except newlines) in filenames. Except that ffmpeg is sharing stdin with while, which can cause some very funky problems. This is safer:

find ... | while read FOO; do echo | (
    ...
); done

and while you’re at it, you might as well take into account the unlikely-but-possible corner case of a filename containing a newline:

find ... -print0 | while read -d $'\0' FOO; do echo | (
    ...
); done

Not literally, of course. This is programming talk, those of you who aren’t programmers can let your eyes glaze over.

I wanted a script to start a bunch of little servers, then wait around for them to finish or when the user interrupts with Ctrl-C, clean up the servers instead of orphaning them. I wanted to propagate the SIGINT to the child processes. I wanted to kill the kids.

The simple way, if you just want to make sure the kids are killed and you don’t care how:

sleep 300 &
# etc.
trap "kill $(echo $(jobs -p)) 2>/dev/null" EXIT
wait

If you only want to trap SIGINT and want to make sure you send SIGINT (not SIGKILL) to the children, then you want to do something like:

trap "kill -INT $(echo $(jobs -p)) 2>/dev/null" INT
wait

Update: I was asked by a shell scripting guru why I needed to do $(echo $(jobs -p)) and not just $(jobs -p). I intended to cover that but forgot. The reason is that $(jobs -p) has newlines and while that’s not usually a problem it is in a trap statement, because it’s evaluated at creation time not at run time. It also means that processes created after you create the trap wouldn’t be killed. Then, he suggested a function instead. Pure brilliance. Where does he come up with these things? Here’s the improved version:

function killkids() { kill $(jobs -p); }
trap "killkids" EXIT

You can still redirect stderr if you want to, but the reason I was directing stderr was because some of the kids may have already died (early evaluation remember) and then kill would needlessly complain. This way, it kills all the kids that are still alive, none more none less.

Have you ever been frustrated with perfectly valid bash syntax being highlighted as incorrect in vim? Like this:

#!/bin/sh
foo=$(ls /tmp)

$() is a perfectly valid, nay preferred substitute for backticks. The problem is that vim is deciding this is pure old bourne shell instead of whatever else we’d like it to be. If you change the shebang to #!/bin/bash and re-edit the file, then the error markings go away. But maybe you’re writing for the nebulous POSIX shell, not bash nor sh. Or maybe you just don’t care and you don’t want vim complaining that you’re using bashisms even though your shebang says sh. You can set the defaults so that it reflects your system and preferences. It’s all there in :help sh.vim.

I’ve occasionally needed to pipeline some processes in Ruby, especially when doing sysadmin-type tasks. You can always do:

system "foo | bar"

but that gets interpreted by the shell. That can be a problem when you need to
include filenames that could have all kinds of nastiness including spaces,
semicolons, exclamation points, etc. You should be shaking in your boots about
now.

Well, for single processes, it’s easy:

system "foo", arg1, arg2

But if you want to pipe them together it’s a bit more difficult. I came up with
this solution, and it seems to be working well in my “convert my cool music
collection into a lame (wink) format my car stereo can understand” script:

  def pipeline(*args)
    pids, r = args.inject([[],nil]) do |memo, cmd|
      pids, rr = memo
      r,w = IO.pipe
      if (pid = fork)
        pids.push pid
        rr.close if rr
        w.close
        [pids, r]
      else
        r.close
        $stdin.reopen rr if rr
        $stdout.reopen w
        cmd = [cmd] if String === cmd
        exec *cmd
      end
    end
    r.close
    pids.each {|p| Process.waitpid(p)}
  end

  pipeline ['oggdec','-o','-',oldpath],['lame','-',newpath]

Do take the time to figure out what’s going on, but don’t do it without a piece
of paper handy or you might sprain something.

Update

I had an error in the original post—if you don’t close rr in the parent of the fork you will end up with too many open files eventually.

Once upon a time I fought the good battle trying to get one of [gkwx]dm to run
a login shell, because you know, we’re logging in. I wanted it to run a
login shell so I could run keychain
or something.

Well today I wanted to figure out how to keep startx from running keychain.
It shouldn’t have been running keychain because keychain runs from
.bash_profile which shouldn’t be sourced for two good reasons: A) it’s not a
login shell (this is a boot script to start freevo), and B) it’s not bash,
it’s /bin/sh (yes, I know that’s really bash but when invoked as sh it doesn’t
source .bash_profile).

It turns out some bright kid created /etc/X11/Xsession.d/91source_profile (I
think that’s what it was called), that sourced every profile file it could
think of whenever Xsession was run. This is both bad and stupid. It’s bad
because sometimes an X session is not a login. It’s stupid because they could
have achieved the same thing in a more elegant way by adding --login to the
shebang line of /etc/X11/Xsession. The place to do login shell stuff is where
you actually log in, i.e. [xgkw]dm, not Xsession.

Please, my friends, if you go out into the world and work on something that
involves login and environment, read the manpages and understand what a login
shell, an interactive shell, and a noninteractive nonlogin shell are and when
you want each. Thanks. End of rant.