The Scoop

For more details on Represent, see our post on the NYT’s Open blog.

Today (well, technically last night) at work we launched a beta Web application called Represent that helps New York City residents keep tabs on what their elected officials are doing. It’s the product of an idea that my colleague Andrei Scheinkman suggested when we sat down in the late Spring to kick around ideas for the Times’ internal technology challenge. We ended up being a finalist with our entry, which became the app you see today, thanks to design work by Stephan Weitberg and advice from a lot of folks.

Since Andrei and I both liked Python, and the contest asked for a working prototype, we built Represent using GeoDjango, the part of Django that makes spatial work easy enough that even I can manage it. This is my first time really dealing with GIS data, and probably the hardest part was getting all the dependencies installed the first time (after you do it once, the install becomes fairly routine, and on the latest Ubuntu version it’s simple).

So if you’re thinking about going the GeoDjango route, or trying it out, I’d offer these suggestions:

• Go with Ubuntu Linux as your OS. It’s just easier. You can run GeoDjango on Windows, but I think I hate the Postgres Windows install and configure process more than just about anything I’ve tried to do on Windows.
• When installing, do what the docs say. These are written by people who have already installed it. Once you start freelancing, your chance of success goes down dramatically.
• Make sure that when you install Postgres, you test it out via the psql shell.
• When in doubt, try the #geodjango IRC channel on FreeNode
• When things just work, do not be surprised or await a message saying they worked.

Once you get GeoDjango installed - and it might take awhile the first time - you’ll be really impressed with how easy it is to use. I’ve written before about the power of LayerMapping, which is a huge gift to people who want to move from using Shapefiles to using GeoDjango. And there are other goodies, too, like the ability to plug into the GeoIP library to place your site visitors on a map (or configure their content based on that information). For Represent, we needed to produce KML files of each district that would display on the results pages. Turns out GeoDjango does that, too. To my eternal shame, I never really got the hang of working with Arc. Now I have a tool that suits me better.

Update: The Miami Herald’s obituary for Liz.

Liz Donovan, one of the best news researchers ever, died of lung cancer in Gainesville, Ga. Truly a pioneer among news librarians, she was instrumental in everything from research for Woodward and Bernstein during Watergate to embracing (and leading other researchers to adopt) the use of blogs, wikis and other new technologies in newsrooms. She worked at the Washington Post and Miami Herald before retiring to the mountains of North Carolina where she continued to learn and spread her knowledge (and some beautiful photographs of the area).

Liz and Margot Williams were the first to welcome me to the News Division and to encourage me to participate in conferences, the mailing list and other activities. Liz encouraged me to write about the technological challenges and opportunities faced by newsrooms, and was generous to a fault with her colleagues seeking help. She was a constant innovator, a skilled researcher and a wonderful person. Among the people in journalism, it is rare to find someone about whom no one has a bad thing to say. Liz was one of those people. I’ll miss her friendship, advice and leadership.

Since I started using Ruby and Rails last year, I’ve come to know and like Capistrano, the deployment utility that makes pushing code and databases around a lot easier than it has to be. We use Capistrano for internal and external deploys at the Times, and it’s a pretty robust tool. While working on Fumblerooski and some other projects, I was looking for a Python-based deployment utility (although, to be fair, Capistrano can be used with just about anything), and I came upon Fabric.

If you haven’t heard of Capistrano, it and similar utilities save you the trouble of typing a series of commands to login to remote servers, update your code, bounce Apache, clean up files, etc. You put all that stuff into a single file that just runs those tasks when you need them. If you have an app that you’re regularly updating, it’s a godsend.

If you’ve used Capistrano before, Fabric is pretty similar in terms of its philosophy; there’s a fabfile in your project (like a capfile for Capistrano) that contains the configuration details and your commands for deploying code, restarting servers and the like. What I like about Fabric is that it makes virtually no assumptions about what you want to do. It doesn’t default to Subversion, or git, as a VCS. It doesn’t assume much of anything, which can be a little daunting if you’re used to a bit of code generation.

But that’s actually a good thing, because you end up writing fabfiles that only contain the stuff you need to have and nothing else. There’s a Django app that ties Fabric more tightly to Django’s manage.py utility, but I’m not sure I see the need for it. Without it, you type something like “fab deploy” instead of “python manage.py fab deploy”. But it’s there.

One thing I’d like to see from Fabric is better handling for sudo tasks on remote servers. This has been raised on the mailing list, and hopefully it gets resolved soon. Otherwise I’m really enjoying its simplicity and will use it for my Django apps going forward.

Until I finish setting up a blog on Fumblerooski, here’s another update on the site’s development:

I’ve added team and player statistics for games in 2006, 2007 and 2008 (example), and will build out individual game stats back to 2001 (example). I’ve fixed several outstanding data issues, including bowl games, overtime games and at the suggestion of Chris McComas, moved conference affiliation from the College model to the CollegeYear model. I’ve added an ever-diminishing list of unbeaten teams.

There is, of course, so much more to do. I’ve got a list of feature ideas a mile long, plus an urgent need for some graphics (using the Google Chart API), team rankings and poll data. But now that the elections are over, I can spend some time working on the site. My long-term goal is to have Fumblerooski be a resource for people seeking to identify that one stat, to study a trend (either on the site or via APIs) or just to lose themselves in college football data. What would you like to see?

Ok, I love the fact that CBS Radio’s Mark Knoller keeps such good tabs on presidential travel, but can somebody please come up with a backup plan in case, heaven forbid, Knoller gets hit by a bus or something? Is this too much to ask of WH reporters? This is the kind of thing where somebody out there is going to do a better job of gathering and displaying this data on the Web. And it should be us, cause it’s our job.