The Fugue Counterpoint by Hans Fugal


Pipelining Processes in Ruby

I've occasionally needed to pipeline some processes in Ruby, especially when doing sysadmin-type tasks. You can always do:

system "foo | bar"

but that gets interpreted by the shell. That can be a problem when you need to
include filenames that could have all kinds of nastiness including spaces,
semicolons, exclamation points, etc. You should be shaking in your boots about

Well, for single processes, it's easy:

system "foo", arg1, arg2

But if you want to pipe them together it's a bit more difficult. I came up with
this solution, and it seems to be working well in my "convert my cool music
collection into a lame (wink) format my car stereo can understand" script:

  def pipeline(*args)
    pids, r = args.inject([[],nil]) do |memo, cmd|
      pids, rr = memo
      r,w = IO.pipe
      if (pid = fork)
        pids.push pid
        rr.close if rr
        [pids, r]
        $stdin.reopen rr if rr
        $stdout.reopen w
        cmd = [cmd] if String === cmd
        exec *cmd
    pids.each {|p| Process.waitpid(p)}

  pipeline ['oggdec','-o','-',oldpath],['lame','-',newpath]

Do take the time to figure out what's going on, but don't do it without a piece
of paper handy or you might sprain something.


I had an error in the original post—if you don't close rr in the parent of the fork you will end up with too many open files eventually.

Comments (2) Trackbacks (0)
  1. If you want to look for some inspiration for metaprogramming constructs to do nifty shell-like things, look no further than scsh, the Scheme shell. It also has the distinction of having the most hilarious Acknowledgements section of any reference manual I’ve ever seen.

    To run the equivalent of “foo | bar “, you’d do (| (foo arg1 arg2) (bar arg1 arg2)). Your syntax seems pretty similar, except you have to manually parse the command lines into strings. That’s just the tip of the scsh iceberg, though, so I recommend taking a look at at least the Process Forms section of the manual for inspiration.

  2. Hiya, never used fork with Ruby before but I’ve done something similiar in Perl.

    my $pid;
    if ($pid = fork ) {
    … do stuff …
    } elsif (defined($pid) ) {
    … do more stuff …
    } elsif($! == EAGAIN) {
    … sleep and redo FORK …
    } else {
    … could not fork …

    Don’t know if this will help you debug the open files …


Leave a comment

No trackbacks yet.