Keep an eye on Ferrett
By Paul Overberg USA Today
A new tool for digging into big federal data sets shows promise, and it's useful now, even in its buggy beta phase. It's the new version of Ferret (Federal Electronic Research and Extraction Tool), renamed Ferrett (http://ferret.bls.census.gov/java/netscape.htm). Built in Java and available free to anyone with a Web browser, it lets users tap microdata-individual records-federal economic, demographic and health surveys. Let's say you want to know whether registration and voting rates for 60- to 65-year-olds are different from those of 65- to 69-year-olds. Or how the age breakdown of union members compares with that of workers covered by union contracts.
"Our goal is to develop an expert-system-type approach about the data," opening it to new audiences, said Cavan Capps, Census Bureau Ferrett project leader.
Or the workforce distribution of your state's foreign-born population.
You can get all this from the Current Population Survey, a monthly 45,000-household sample that yields a wealth of annual demographic data snapshots in addition to its main job, producing the monthly unemployment rate. But only a few people know how to wade through the survey's hierarchical file structure, various weightings for household, family and person records, top-coded variables and other subtleties.
In its infancy
Before the Web, using CPS required buying computer tapes and mastering arcane concepts with codebook and phone help. Ferret 1.0, released three years ago, simplified the task tremendously. It offered a Web interface to build queries, and also allowed users to slice out just the variables they wanted to download. It carried pretty good online documentation, but it still required subject familiarity.
The newest version is better. It's been renamed Ferrett to avoid trademark entanglements. It walks a user through the process of picking a survey and variables. You can also check off which values you want from each variable. Finally, Ferrett allows on-screen tabulation or download.
Once you've chosen the survey and variables, arranging them is much like using Excel's pivot table wizard. An animated graphic of a ferret wearing a mortarboard demonstrates.
It's not without problems. It's slow. A memory leak causes it to occasionally forget where you were in building a query. Some of the point-and-click steps require a second or third try before they're executed. Windows sometimes take three tries to close. The navigation is pretty good, but not great.
But even with all that, Ferrett's power makes it worthwhile if you need to slice up the CPS for answers that go beyond published tables.
Future potential
Ferrett's real potential lies in its promise to open up other government datasets over an effort called the Data Web. The National Center for Health Statistics has helped with Data Ferrett's development, and some of its surveys should be available soon. Among them:
National Health and Nutrition Examination Survey (HANES) National Health Interview Survey (NHIS) National Ambulatory Medical Care Survey (NAMCS)
Other Census-data sets that may soon be added include annual population estimates, annual County Business Patterns and the 1997 Economic Census.
Even state and local data sets may become available, said Cavan Capps, the Census Bureau project leader. Already, Ferrett taps 350 gigabytes of data on computers scattered around the country. "A Web browser is designed to be a document browser, but then you happen on data," Capps said. "Ferrett is designed to be a data browser."
Experts to audience
Capps said that only a handful of labor economists and demographers understand CPS concepts, variables and weights and how to link them in time series to measure change. Building the rules that experts know into Ferrett, Capps said his team hopes to make the data self-aware, and useful to new audiences.
One example is state-level CPS data. Each March, the CPS panel is asked a host of questions on household members and their income sources the previous year.
The panel typically doesn't include enough people in each state to produce sound results. But two years of March data does, and that's how Census experts use it to publish reports. Capps hopes Ferrett will be able to understand such rules and make them transparent.
"Our goal is to develop an expert-system-type approach about the data," opening it to new audiences, he said.
In addition to the March demographic and income questions, the CPS each October carries school enrollment questions, and every other November surveys voting and registration in the just-past national election. Periodic supplements address computer usage, fertility, marital history, displaced and contingent workers and job tenure.
An example
A simple example might help illustrate. I entered Ferrett and logged in with my e-mail address. (If you can't trust the Census Bureau's promise to never disclose it, who can you trust?) On the introductory screen, only one option's available. Clicking on that button opens Ferrett's tabbed palette. A Windows-style left pane allows browsing and selection of the survey you need.
I chose the March 1999 CPS and searched for "age" and "union." Ten variables came back. I selected two and clicked on the "Browse variable" button at top right.
That gave me a screen where the variable's details were laid out, and check boxes allow the option of filtering by certain values. Selecting a variable there pushes it into the Data Basket.
Moving on to the "Get Data" tab, the option is to "Extract" (download) or "Tabulate."
Clicking on the latter opens a screen where the selected variables can be dragged and dropped as row and column variables, just like Excel's pivot table wizard.
Clicking the green "Go" button launches the process. Going back and choosing "Extract" produces a download option.
Ferrett automatically weighted the tabulation so it got national totals, not just survey respondent totals. It also added a MARSUPWT variable to the download.
This is the weighting by which I'd multiply each record by to produce the national total. I didn't have to specify this, Ferrett just knew.
This tool isn't ready for quick answers or deadline work. A query can take several tries, Capps admits. Ferrett should come out of beta this fall.
Paul Overberg can be reached by e-mail at poverberg@usatoday.com