The Fugue

So I finally got around to trying out the "New FamilySearch" today. I am both impressed and disappointed.

First the good parts. NFS (you didn't think I was going to type "New FamilySearch" over and over, did you?) has an impressive goal and paradigm. The goal is to create one hugemongous centralized database for all church members. The idea is to get away from the half dozen church databases (Ancestral File, IGI, etc.), and the half gazillion individual databases. A noble goal but a very scary one. It would be easy to screw this up and make a bigger mess than that with which we started. In fact this is why I have been reluctant to check it out—I didn't want to be disappointed and I wholly expected to be.

Well, they actually pull it off quite well. The new paradigm is to keep everything and to promote recording evidence. In short, genealogy done right. When you merge a person, that is recorded and available for others to see. When you want to change information, you don't change it directly (as you would in a conclusion-based program like PAF), but instead you "add an opinion" complete with sources and/or notes. If you think that a piece of information is wrong and you have evidence against it, you can dispute it (again, giving source and notes). The old "wrong" information isn't eliminated, but it is marked as disputed. The changes and choices you make about people show up in the pedigree chart etc. This is multi-user genealogy done well (I might call it "distributed genealogy", but I'll reserve that term for something better, as you'll see later).

From the perspective of an LDS member this is a fantastic system. When ordinances are performed in the temple they are immediately reflected in the database. When you want to do temple work for so-and-so, you state your intentions in the system and print out a page to take to the temple with you. If anyone else tried to do the same work, they'd see it was in process. This will drastically reduce—perhaps even essentially eliminate—duplicated effort in the temple. I have to say it's about time. It would have been cool 10 years ago. It was expected 7 years ago. Now it's finally here.

There are some other cool tidbits, too, like the pedigree view which combines couples to make better use of space (are they the first to think of it? Probably not, though I haven't personally seen this approach before):

There's an info box at the bottom with different tabs, one of which is "possible duplicates". I much much much prefer working with duplicates in this manner, rather than a global "match and merge". Very nice. There are also time lines and Google Maps integration (see where your ancestor was born, married, died, etc.). And those little temple icons unobtrusively notify you of potential temple work to do (or that has already been done). Overall they make nice use of AJAX, too.

But there's problems. Big problems.

It's slow. Painfully slow. It's slow enough to be a real pain for doing actual genealogical work. Maybe people with limited computer skills wouldn't find it slow, because it moves at about the pace they can keep up with. But for those of us in the computer age (read: almost everyone in my generation or younger) it is painful and restrictive. Why is it slow? Because it's a web app. News flash! Even AJAX web apps are slow.

Ok, it's slow. No big deal, right? Just download the GEDCOM, do your research, and upload the changes. Right? I have news for you. There's no exporting data from NFS. The help center has this to say:

Exporting Information from FamilySearch for Use in Your Personal Computer

This topic describes how to get information from FamilySearch into your family history computer program.

If you find information in FamilySearch that you do not have, you will need to either use the cut and paste features of your operating system or retype it into your computer program.

Currently, FamilySearch does not support downloading information for use with Personal Ancestral File or similar computer programs. Family history computer programs may choose to support this feature when it becomes available from FamilySearch.

Really. Cut and paste! It is a big black hole waiting to consume your information and display it to you on its terms only. Its slow terms. You want to make a family pedigree website? Write a script to spit out all the place names of your ancestors so you can put blue dots on a map? Make a Google Maps mashup? Do any number of other useful things with a GEDCOM export, including actually be able to work with it at a reasonable speed, put it on your handheld for reference at the family history library? Print out reports? No way. Uh-uh. Remember how I avoided using the term "distributed genealogy"? It's like having your genealogy in a distributed revision control system like mercurial or git, but you can only access the one single repository with a web interface. You can't check out the code. You can't work offline. You can't use your own tools. You can't write emergent scripts. You're screwed.

For understandable reasons, you can't see information on living people, and they don't show up as search results. You do get access to your own ancestors and descendents and your spouse, but apparently not your spouse's family, your siblings, or any information on living people (like your parents' birthdays, etc.). You can enter this information in, or upload it in a GEDCOM. But the first rule of genealogy is start with your 4 generations. If everyone starts with their 4 generations, but most of those people are still alive, then how much effort is duplicated? How many duplicate versions of my dad will there be? Well let's see, he has 11 siblings, various aunts and uncles who are into genealogy, 7 children (who should all see the same record, but might conceivably enter conflicting information). Not a huge problem, but an annoyance. Once you fill out the tree to the dead people (hint: upload a GEDCOM of what you already have here, but only those first couple generations), then you find and link the dead people into the tree, then you have a nice resource. So far, it's just a research resource—I wouldn't trust a lot of things further than I can throw them, but they make good research jumping-off points. Maybe eventually through the hard work of thousands it will converge to a respectable database, in the spirit of a wiki.

Also, it's presently restricted to LDS members (you need your membership number and confirmation date to register). The best genealogists I know aren't LDS. Certainly the bulk of decent genealogists I know aren't LDS. Most of the lousy genealogists I know are LDS. (Of course, that doesn't mean we have a monopoly on lousy genealogists, I just haven't had reason or opportunity to mingle with lousy non-LDS genealogists much). So this seems like a drawback across the board.

Maybe down the road (I think it's still beta, though they never use that word) it will allow GEDCOM export and be available to all genealogists. Maybe the speed issue will be addressed, or they'll come up with a desktop client. Maybe this will be the rockingest genealogy database ever. Or maybe it will be of marginal interest—a great way to prepare names for the temple and avoid duplicate temple work, but not a good tool for daily genealogical work. Time will tell.

I am impressed by the no-information-loss implementation. I'd like to propose taking it a step further. What if we could publish genealogical repositories on websites like we do with mercurial or git? What if we had the genealogical equivalent of github? What if you and all the other genealogists out there could, without information loss, match and merge and add information and correct information and unmerge faulty merges and… all without loss of information, the ability to go back in time (like you can with a revision control system), etc. A global genealogical database, a global record of genealogical discovery. Now, one huge database doesn't make a lot of sense. It'd be a pain to push and pull. So you'd have to be able to push and pull only pieces of the tree. And of course the merging, confidence, dispute, etc. aspects would have to be dealt with well (as they mostly are in NFS, though there would be unique challenges for it in a truly distributed genealogical system). Just imagine the potential. And feel free to expound on your imaginings in the comments.

So you slept through a few sessions of conference (or you were distracted by the computer or stuck in the bathroom with a potty training toddler). The natural thing to do is to listen to sessions one at a time on your mp3 player over the next week or two. But it would be too much work to go to the archives website every day to download the appropriate session, then sync your mp3 player. No, you need to download them all now. But clicking on each one is tedious at best.

Enter a script. Now I've done something like this in the past with just sed and grep, but this time I thought I'd skip the sed trial and error and practice some Hpricot. It was quick and painless:

require 'rubygems'
require 'hpricot'
require 'open-uri'
CONFERENCE_URL="http://www.lds.org/conference/sessions/display/0,5239,49-1-851,00.html"
doc = Hpricot(open(CONFERENCE_URL))
puts (doc/"a").map{|a| a['href']}.join("\n")

That little script extracts all the urls from a URL. Well maybe there's a program that does that already. Maybe it's urlview, but an impatient google didn't find it in time (and urlview isn't installed on my laptop). Now you continue on with the grep magic:

./strip_urls.rb | grep mp3 | grep -v Complete | xargs wget

And you're on your way.