The Fugue

A friend of mine is struggling to grok the MVC pattern. I remember when I first tried to grok it how frustrating it was. There wasn’t one final “a-ha” moment when I grokked it, but one day I looked back and realized I understood it. At that moment it dawned on me just how amazingly silly this whole process is. MVC is not some magical formula that if you implement it will endow your application with magical powers. It is a paradigm. A worldview of application programming. Once you get it, you realize that everything is MVC. It’s just that some of it is cleaner MVC than others. The trick is in keeping the three components separate, but they’re always there. My friend is still confused after an IM conversation, because his preconceptions about the computer and probably his desire to find that something different are getting in the way. So let’s use a non-computer analogy.

You’ve seen The Hitchhiker’s Guide to the Galaxy, right? Of course you have. If not, I’ll wait.

Ok, remember the scene where Trillian is at a desk with a Vogon who asks her here home planet? She says it’s Earth, the Vogon can’t find Earth, she tells her where it is, she says it was destroyed, and does Trillian perhaps have another home planet? Yeah, I don’t do it justice. Ok, so Trillian is the user. The Vogon is the controller, the computer is the database, and there are several views in play. The first view is the Vogon telling Trillian that it was destroyed. The second view is when she turns the computer monitor around so the incredulous Trillian can see for herself the video of the Earth being blown up. The third view is authorization with Zaphod’s autograph.

The nice thing about MVC, see, is that the model has no clue about the view or the controller. It just provides an interface for the controller to get and change the model’s information. Likewise the view is pretty dumb. It accesses the model, or a snapshot thereof. But it doesn’t make decisions. It just displays stuff. The controller is the “brains” behind the operation. It coordinates showing the user the correct view of the bit of the model that the user wants to see. It handles updating the model with new information.

Ok, still confused? Let’s try another. This time the controller is your CD player. The model is a CD. The view is your amplifier and speakers, or whatever else you have plugged into the audio outputs. An oscilloscope maybe? You the user push buttons and the controller changes the view. The model and view are obviously decoupled. But this analogy isn’t perfect because the view isn’t accessing the model directly, as is usually the case in MVC, but is being spoonfed the data by the controller. In my opinion this distinction is irrelevant, because the whole point is to achieve decoupling and whether the controller provides the decoupleable data or the view fetches it directly from the model is irrelevant to decoupleability, as we can see with this real-world example. Also this analogy might be better if we think back to cassette tapes which allow us to change the model (by recording), whereas the CD is read-only.

Ok, so now let’s bring it to the computer domain. The model is a database. Actually it’s usually a wrapper around a database (which is in turn usually decoupled too), like ActiveRecord or your homegrown classes. You almost always write classes, because representing “stuff” is what OOP is good at. It doesn’t have to be a real database of course, it could be just objects in RAM or a flat file. The key word for models is data.

Now, the view is almost always entirely contained in the GUI toolkit you’re using. The toolkit writers figured out all that boring stuff that lets you draw widgets and pack them together in a dialog box, etc. All that stuff you wouldn’t want to do manually is the view. You may have some view code which deals with setting up the widgets, etc. The view may not even be visual—it might be a file on disk or sound coming out the speakers. The key word for views is render.

The controller is everything else. Those GUI toolkits usually have some controller help too, since GUIs are interactive views and the events that the controller reacts to are coming from the view, in a sense. When you write callbacks, you’re writing the controller. The controller figures out what to do in response to various events. It decides how to manipulate the views. It tells the views what data to display. The key word is event, and maybe coordination.

So you see, if you’re using a toolkit or framework you’re already sort of doing MVC. The real litmus test can be boiled down to the following questions. If you can answer yes to all of them, then you’re doing MVC. If not, you’re not.

1. Could I render different views without duplicating control code?
2. Could I use my model code unchanged in a completely different application, and could I switch databases (if you have a persistent model) like I switch CDs?
3. Is my model ignorant of the rest of the program?
4. Do I feel like MVC is making my life easier, not harder?
5. Have I sent chocolate to Hans recently?

There you have it. I hope it was helpful.

I was doing some maintenance on my blog, and was devastated to find that Typo was taking 225 megabytes of resident RAM. Yikes! After some creative debug thinking and digging I figured out it was due to sessions. Typo now stores sessions in the database, so my maintenance cron job to delete old sessions didn’t clean up old sessions. (Ha! had you going for a second!)

Well I could write a cron job to run a script to clean the sessions out of the db, like:

#!/bin/sh
sqlite3 /path/to/typo/db/production.db 'delete from sessions'

Ok, that’s a bit extreme, but you get the idea. But when I deleted the sessions in this manner the memory usage didn’t drop at all until I had restarted the server, which seems unnecessary. So instead I changed typo’s configuration to use a different session store. I commented out this line in config/environment.rb:

-  config.action_controller.session_store = :active_record_store
+  #config.action_controller.session_store = :active_record_store

Then I restarted the server and fired up a browser. “Huh, that’s odd… no sessions in tmp/sessions or /tmp or anywhere I can see. No, they’re not in the database…” What I was seeing didn’t match up with what all the stuff Google said. The default session store was PStore, aka file system, so they said. But apparently that recently changed in Rails, and now the default is CookieStore. From ActionController::Base documentation:

Sessions are stored in a browser cookie that‘s cryptographically signed, but unencrypted, by default. This prevents the user from tampering with the session but also allows him to see its contents.

Do not put secret information in session!

Well a quick grep -ri session app lib told me that typo wasn’t storing
anything secret, so I decided that default was alright with me. Now I don’t
have to set up any session cleanup script at all. Sweet.

Now, don’t stop there. You should set your session key and secret while you’re
hanging out in config/environment.rb. Add the following lines in the same
place as the line you commented out above:

config.action_controller.session['session_key'] = 'something unique'
config.action_controller.session['secret'] = 'get this from rake secret'

I have a little rails app that can have a lot of
images on each page. It worked fine in development, but when I went to deploy
it with mongrel behind apache’s mod_proxy, it started behaving oddly. It would
randomly and inexplicably not serve up thumbnails. I checked all my code, the
thumbnails were being generated properly, and my code checked out. Apache would
complain with something like this:

[error] proxy: error reading status line from remote server 127.0.0.1, referer: http://foton.fugal.net/album/detail/23
[error] proxy: Error reading from remote server returned by /foto/thumbnail/93, referer: http://foton.fugal.net/album/detail/23

Carefully watching my rails logs, I noticed they weren’t mentioning the
offending files (that Apache was complaining about), so I guessed the problem
must be between Apache and my code, i.e. in Mongrel. So I fired it up with
Lighttpd instead of Mongrel, and lo and behold it worked fine. To be fair, I
didn’t try upgrading Mongrel (from 1.0.1 to whatever’s current—1.1 I think).
Maybe this is a fixed bug. But now you know it’s Mongrel’s fault, if you’re
seeing something similar.

I’m writing a little photo gallery of my own, because everything out there stinks. But sending big images files in Rails (using send_file and send_data) is slow, mostly because you tie up a whole rails process just feeding data to the web. Web servers like Apache, Lighttpd, and Mongrel are good at serving static files, let them do it.

That’s the idea behind X-Sendfile. If you send an X-Sendfile header with the path of the file you want to send, then a supporting webserver will do the dirty work and do it fast, and you can get on with serving other requests.

That’s the theory anyway, but there’s some bumps in the road. First, AFAICT
mongrel doesn’t support X-Sendfile. This is fine when mongrel is running behind
an Apache proxy which does, but kind of throws a wet blanket on development and
apachephobes like myself. Ok, apachephobe might be a bit strong, but I don’t
want to set that monster on my laptop just for some rails development. So mongrel’s out. Correct me if I’m wrong.

Lighttpd supposedly invented X-Sendfile, but 1.4.x and earlier don’t seem to
support it. Instead, you have to use the header X-LIGHTTPD-send-file. Also, it
doesn’t work unless Content-Length is properly set (or perhaps if it’s absent).
This is bad news for rails users, since a bug in rails causes the
Content-Length header to be set to the content, which is not the file. If you
do render :nothing => true, then the content is one space character, and the
Content-Length is 1, and Lighttpd defiantly refuses to fix it. So you either
have to work around the rails bug, or upgrade to lighttpd version 1.5.x (now in
release candidate) which supposedly works (I haven’t tested it—I can’t get it
to compile on Leopard). I say bug in rails, but frankly I’m more inclined to
consider this bad behavior on the part of lighttpd. In that vein, here is a
patch for lighttpd version 1.4.18 that will enable both X-LIGHTTPD-send-file
and X-Sendfile headers with rails 1.2.3 which has the Content-Length resetting
behavior. It makes lighttpd set the Content-Length on its own. Thanks to
stbuehler for the patch.

--- src/mod_fastcgi.c.orig      2007-11-05 13:52:47.000000000 -0700
+++ src/mod_fastcgi.c   2007-11-05 13:55:17.000000000 -0700
@@ -2530,22 +2530,28 @@
                }

                if (host->allow_xsendfile &&
-                                   NULL != (ds = (data_string *) array_get_element(con->response.headers, "X-LIGHTTPD-send-file"))) {
+                                   ((NULL != (ds = (data_string *) array_get_element(con->response.headers, "X-LIGHTTPD-send-file")))
+                                     || (NULL != (ds = (data_string *) array_get_element(con->response.headers, "X-Sendfile"))))) {
                    stat_cache_entry *sce;

                                         if (HANDLER_ERROR != stat_cache_get_entry(srv, con, ds->value, &sce)) {
-                                               /* found */
-                                                con->parsed_response &= ~HTTP_CONTENT_LENGTH;
-
+                                               data_string *dcls = data_string_init();
+                                                /* found */
                        http_chunk_append_file(srv, con, ds->value, 0, sce->st.st_size);
                        hctx->send_content_body = 0; /* ignore the content */
                        joblist_append(srv, con);
-                                       }
-                                        else
-                                        {
-                                               log_error_write(srv, __FILE__, __LINE__, "sb",
-                                                       "send-file error: couldn't get stat_cache entry for:",
-                                                       ds->value);
+
+                                               buffer_copy_string_len(dcls->key, "Content-Length", sizeof("Content-Length")-1);
+                                               buffer_copy_long(dcls->value, sce->st.st_size);
+                                               dcls = (data_string*) array_replace(con->response.headers, (data_unset *)dcls);
+                                               if (dcls) dcls->free((data_unset*)dcls);
+
+                                               con->parsed_response |= HTTP_CONTENT_LENGTH;
+                                               con->response.content_length = sce->st.st_size;
+                                       } else {
+                                               log_error_write(srv, __FILE__, __LINE__, "sb",
+                                                       "send-file error: couldn't get stat_cache entry for:",
+                                                       ds->value);
                                         }
                }
--- src/response.c.orig 2007-11-05 14:08:26.000000000 -0700
+++ src/response.c      2007-11-05 14:04:49.000000000 -0700
@@ -59,7 +59,8 @@
    ds = (data_string *)con->response.headers->data[i];

    if (ds->value->used && ds->key->used &&
-                   0 != strncmp(ds->key->ptr, "X-LIGHTTPD-", sizeof("X-LIGHTTPD-") - 1)) {
+                   0 != strncmp(ds->key->ptr, "X-LIGHTTPD-", sizeof("X-LIGHTTPD-") - 1) &&
+                   0 != strncmp(ds->key->ptr, "X-Sendfile", sizeof("X-Sendfile") - 1)) {
            if (buffer_is_equal_string(ds->key, CONST_STR_LEN("Date"))) have_date = 1;
            if (buffer_is_equal_string(ds->key, CONST_STR_LEN("Server"))) have_server = 1;

Then, you need to configure your lighttpd server. Run script/server lighttpd once to generate config/lighttpd.conf, and add this bit to the fastcgi.server section:

    "allow-x-send-file" => "enable"

Finally, use it—either by setting the X-Sendfile header manually or by using the rails x_send_file plugin (I recommend the latter).

Here’s some links for more reading:

• HOWTO Send Files Fast
• Send files faster with X-Sendfile
• X-Sendfile (for lighttpd 1.4.x—remember the correct Content-Length requirement and that Rails doesn’t)
• mod_proxy_core got X-Sendfile support (for lighttpd 1.5.x—this looks really nice)
• The Rails “bug” that ruined X-Sendfile on lighttpd 1.4.x
• mod_xsendfile for Apache 2 (I haven’t used it, but I assume it works fine)

I am in the process of migrating my blog to Typo. Blosxom has served me well, but I think it’s time I joined the ranks of cool blogs and allowed people to comment, trackback, etc. Blosxom can do all that, but it’s not very much fun.

Blosxom’s strength is its simplicity. When I wanted a simple blog, it was perfect.

Now, getting typo to work was not a walk in the park. I’ll walk you through what I had to do.

First, you have to get typo and install it. I hear rumors that version 4 release is imminent, so I grabbed the svn version. Next, make a database and set up config/database.yml. Now you can run script/server -e production and point your browser at http://localhost:3000/ and give it a whirl.

I tried the RSS and atom converters in db/converters but I wasn’t satisfied with the results. So I wrote a blosxom converter, which I will contribute. If you can’t find it feel free to contact me.

The hard part was getting typo to run under a subdirectory in my mixed apache/lighttpd setup. All fugal.net sites are running through Apache. My rails apps run through lighttpd, with apache’s mod_proxy. So, you have to configure apache, lighttpd, and rails each in turn.

Apache Configuration

    ProxyPass               /typo http://127.0.0.1:81/typo
    ProxyPassReverse        /typo http://127.0.0.1:81/typo
    <Proxy *>
        Order allow,deny
        Allow from all
    </Proxy>

Lighttpd Configuration

The normal lighttpd stuff, and then:

$HTTP["url"] =~ "^/typo(/|$)" {
    server.indexfiles = ("dispatch.fcgi")
    server.document-root = "/srv/www/typo/public/"
    server.error-handler-404 = "/dispatch.fcgi"
    server.errorlog = "/srv/www/typo/log/error.log"
    accesslog.filename = "/srv/www/typo/log/access.log"
    fastcgi.server = ( ".fcgi" => (
        "typo" => ( "min-procs" => 1, "max-procs" => 1,
            "socket" => "/tmp/typo.fcgi.socket",
            "bin-path" => "/srv/www/typo/public/dispatch.fcgi",
            "bin-environment" => ( "RAILS_ENV" => "production" ),
            "idle-timeout" => 120
            )
        )
    )
}

You don’t need strip-request-uri, because we’ll take care of the typo/ prefix in rails.

Rails Config

Add this to config/environment.rb:

    ActionController::AbstractRequest.relative_url_root = "/typo"

Finally, and this was the real stink, you have to help lighttpd find files in public/. Lighttpd gets a request for http://hans.fugal.net/typo/foo.html which is a static file in public/, and so it looks in the document root to find /typo/foo.html, which means it looks for /srv/www/typo/public/typo/foo.html, which doesn’t exist. This is easy to fix:

    cd /srv/www/typo/public
    ln -s . typo

et voilá! You should be up and running.

For more reading see http://znark.com/blog/articles/2005/12/11/got-typo-working and http://blog.lighttpd.net/articles/2005/11/23/lighttpd-1-4-8-and-multiple-rails-apps.