The Fugue Counterpoint by Hans Fugal

5Nov/071

X-Sendfile

I'm writing a little photo gallery of my own, because everything out there stinks. But sending big images files in Rails (using send_file and send_data) is slow, mostly because you tie up a whole rails process just feeding data to the web. Web servers like Apache, Lighttpd, and Mongrel are good at serving static files, let them do it.

That's the idea behind X-Sendfile. If you send an X-Sendfile header with the path of the file you want to send, then a supporting webserver will do the dirty work and do it fast, and you can get on with serving other requests.

That's the theory anyway, but there's some bumps in the road. First, AFAICT
mongrel doesn't support X-Sendfile. This is fine when mongrel is running behind
an Apache proxy which does, but kind of throws a wet blanket on development and
apachephobes like myself. Ok, apachephobe might be a bit strong, but I don't
want to set that monster on my laptop just for some rails development. So mongrel's out. Correct me if I'm wrong.

Lighttpd supposedly invented X-Sendfile, but 1.4.x and earlier don't seem to
support it. Instead, you have to use the header X-LIGHTTPD-send-file. Also, it
doesn't work unless Content-Length is properly set (or perhaps if it's absent).
This is bad news for rails users, since a bug in rails causes the
Content-Length header to be set to the content, which is not the file. If you
do render :nothing => true, then the content is one space character, and the
Content-Length is 1, and Lighttpd defiantly refuses to fix it. So you either
have to work around the rails bug, or upgrade to lighttpd version 1.5.x (now in
release candidate) which supposedly works (I haven't tested it—I can't get it
to compile on Leopard). I say bug in rails, but frankly I'm more inclined to
consider this bad behavior on the part of lighttpd. In that vein, here is a
patch for lighttpd version 1.4.18 that will enable both X-LIGHTTPD-send-file
and X-Sendfile headers with rails 1.2.3 which has the Content-Length resetting
behavior. It makes lighttpd set the Content-Length on its own. Thanks to
stbuehler for the patch.

--- src/mod_fastcgi.c.orig      2007-11-05 13:52:47.000000000 -0700
+++ src/mod_fastcgi.c   2007-11-05 13:55:17.000000000 -0700
@@ -2530,22 +2530,28 @@
                }

                if (host->allow_xsendfile &&
-                                   NULL != (ds = (data_string *) array_get_element(con->response.headers, "X-LIGHTTPD-send-file"))) {
+                                   ((NULL != (ds = (data_string *) array_get_element(con->response.headers, "X-LIGHTTPD-send-file")))
+                                     || (NULL != (ds = (data_string *) array_get_element(con->response.headers, "X-Sendfile"))))) {
                    stat_cache_entry *sce;

                                         if (HANDLER_ERROR != stat_cache_get_entry(srv, con, ds->value, &sce)) {
-                                               /* found */
-                                                con->parsed_response &= ~HTTP_CONTENT_LENGTH;
-
+                                               data_string *dcls = data_string_init();
+                                                /* found */
                        http_chunk_append_file(srv, con, ds->value, 0, sce->st.st_size);
                        hctx->send_content_body = 0; /* ignore the content */
                        joblist_append(srv, con);
-                                       }
-                                        else
-                                        {
-                                               log_error_write(srv, __FILE__, __LINE__, "sb",
-                                                       "send-file error: couldn't get stat_cache entry for:",
-                                                       ds->value);
+
+                                               buffer_copy_string_len(dcls->key, "Content-Length", sizeof("Content-Length")-1);
+                                               buffer_copy_long(dcls->value, sce->st.st_size);
+                                               dcls = (data_string*) array_replace(con->response.headers, (data_unset *)dcls);
+                                               if (dcls) dcls->free((data_unset*)dcls);
+
+                                               con->parsed_response |= HTTP_CONTENT_LENGTH;
+                                               con->response.content_length = sce->st.st_size;
+                                       } else {
+                                               log_error_write(srv, __FILE__, __LINE__, "sb",
+                                                       "send-file error: couldn't get stat_cache entry for:",
+                                                       ds->value);
                                         }
                }
--- src/response.c.orig 2007-11-05 14:08:26.000000000 -0700
+++ src/response.c      2007-11-05 14:04:49.000000000 -0700
@@ -59,7 +59,8 @@
    ds = (data_string *)con->response.headers->data[i];

    if (ds->value->used && ds->key->used &&
-                   0 != strncmp(ds->key->ptr, "X-LIGHTTPD-", sizeof("X-LIGHTTPD-") - 1)) {
+                   0 != strncmp(ds->key->ptr, "X-LIGHTTPD-", sizeof("X-LIGHTTPD-") - 1) &&
+                   0 != strncmp(ds->key->ptr, "X-Sendfile", sizeof("X-Sendfile") - 1)) {
            if (buffer_is_equal_string(ds->key, CONST_STR_LEN("Date"))) have_date = 1;
            if (buffer_is_equal_string(ds->key, CONST_STR_LEN("Server"))) have_server = 1;

Then, you need to configure your lighttpd server. Run script/server lighttpd once to generate config/lighttpd.conf, and add this bit to the fastcgi.server section:

    "allow-x-send-file" => "enable"

Finally, use it—either by setting the X-Sendfile header manually or by using the rails x_send_file plugin (I recommend the latter).

Here's some links for more reading: