{
  "id": "1998-may-car-with-microdata",
  "slug": "1998-may-car-with-microdata",
  "url": "https://dwillis.github.io/uplink/article/1998-may-car-with-microdata.html",
  "year": 1998,
  "month": "May",
  "monthNum": 5,
  "headline": "CAR with microdata",
  "kicker": "STATS FROM THE ROAD",
  "author_name": "Paul Overberg",
  "author_title": "",
  "affiliation": "USA Today",
  "summary": "Microdata refers to individual-level records from surveys that allow journalists to perform unique cross-tabulations and uncover news stories, though working with it requires proper understanding of statistical weights and file structures. The article explains why microdata is valuable compared to summary data and provides guidance on how to properly handle hierarchical survey data like that from the Current Population Survey.",
  "keywords": [
    "microdata",
    "survey",
    "analysis"
  ],
  "topics": [
    "Data & Databases",
    "Investigative Methods",
    "Technology"
  ],
  "technologies": [
    "SPSS",
    "SAS"
  ],
  "full_text": "CAR with microdata\n\nBy Paul Overberg USA Today\n\nMicrodata is a word from the social sciences. Loosely, it means data at the level at which it's collected. More specifically, it often means the individual records that result from a carefully designed survey where the unit is often a single person, household, incident or something similar. Microdata is usually contrasted with macrodata, a generic term for various levels of summarization and cross-tabulation.\n\nWhy bother?\n\nIf you're thinking, \"Microdata! Fancy word for something I already do!\" you're wrong. True microdata can be much more rewarding. You can do cross-tabulations that no one else has ever done - and find news. But it's also trickier. It's often produced by a survey or statistically random selection, so each record represents a larger number of its type in the entire population. But we have to preserve and use each record's weight correctly. Depending on the survey's precision, weights can vary a little or a lot from record to record. It's often collected in a hierarchical file, something akin to interleaving the tables of a relational database. For instance, the Current Population Survey, done monthly by the Census Bureau for the Bureau of Labor Statistics, collects information about 50,000 households into one big file with three record types for their personal, family and household characteristics. Often, a researcher will pull data from the people and household records into a new working file. This is called rectangularizing.\n\nHousehold 1 - all data Family 1 - all data Person 1 - all data Person 2 - all data Person 3 - all data\n\nbecomes in a working file\n\nPerson 1 - some data and Household 1 - some data Person 2 - some data and Household 1 - some data Person 3 - some data and Household 1 - some data\n\nThis turns the hierarchical file into a flat (rectangular) file that's smaller and easier to analyze. If you've ever analyzed the Fatal Accident Reporting System (FARS) data from the National Highway Traffic Safety Administration, you've faced this. For each fatal accident record, there's one or two or more related vehicle records, each of which has one, two or more related person records.\n\nBefore using microdata\n\nDon't - if you don't have to. Often, summary data is all you need. Start there. Read the standard reports published by the keeper of the data. Look in the back for detailed cross-tabulations if the standard summaries aren't enough. For instance, median household income appears right up front in the Census Bureau's annual report on income. Dig a little and you can find median income by race (all whites). Dig deeper and you can find median income by race by age group (whites 45-54). Dig deeper and you can find median income by race by age by education (whites 45-54 with a bachelor's degree), deeper still for median income by race by age by education by work status (whites 45-54, bachelor's degree, work full-time) and even deeper still for median income by race by age by education by work status by region (whites 45-54, bachelor's degree, work full-time, Midwest).\n\nTalk to whoever keeps the data before you spend a week plowing into a microdata maze. They may have done what you're attempting or have good advice on how to do it (or why not to).\n\nLearn the data thoroughly. This is especially true for microdata based on surveys or hierarchical files. Read the appendix on the sampling design. Read the file list and values. Try to duplicate published calculations. Learn SPSS or SAS. Either will do, but knowing neither leaves you fumbling with spreadsheets and database managers. They may work, but they're sort of like using a regular screwdriver on a Phillips head screw: You have to work harder and still may get inferior results.\n\nPaul Overberg can be reached at (703) 276-5427 or by e-mail at poverberg@usatoday.com",
  "provenance": {
    "extracted_by": "anthropic/claude-sonnet-4-6",
    "extracted_at": "2026-07-05",
    "verified": true
  },
  "similar": [
    {
      "slug": "2002-september-october-census-workshops-scheduled-for-reporters-editors",
      "score": 0.209
    },
    {
      "slug": "1999-october-uplink-survey",
      "score": 0.205
    },
    {
      "slug": "2004-january-february-readmetxt",
      "score": 0.189
    },
    {
      "slug": "1995-march-the-survey-says",
      "score": 0.158
    },
    {
      "slug": "2004-january-february-measuring-migration-with-pums-data",
      "score": 0.146
    }
  ],
  "html_url": "https://dwillis.github.io/uplink/article/1998-may-car-with-microdata.html",
  "json_url": "https://dwillis.github.io/uplink/article/1998-may-car-with-microdata.json"
}