{
  "id": "1997-september-testing-for-significance",
  "slug": "1997-september-testing-for-significance",
  "url": "https://dwillis.github.io/uplink/article/1997-september-testing-for-significance.html",
  "year": 1997,
  "month": "September",
  "monthNum": 9,
  "headline": "Testing for significance",
  "kicker": "Stats from the road",
  "author_name": "John Sullivan and Sarah Cohen",
  "author_title": "",
  "affiliation": "NICAR",
  "summary": "Journalists can use statistical significance tests, particularly chi-square tests, to determine whether patterns in data like employment discipline differences represent real trends or random chance when investigating potential discrimination.",
  "keywords": [
    "statistics",
    "significance",
    "discrimination"
  ],
  "topics": [
    "Data & Databases",
    "Investigative Methods",
    "Journalism & Newsroom"
  ],
  "technologies": [
    "SPSS",
    "Excel"
  ],
  "full_text": "Many journalists spend hours staring at data and wondering if they are looking at something that happened by accident. When a sheriff disciplines relatively more female deputies one year than male, is it possible that the pattern is just a fluke? The sheriff will argue it was a random blip in the numbers. A reporter has been hearing of other signs of sex discrimination. A significance test can help guide you toward the truth.\n\nPick your test\n\nMany journalists who look for significance use a chi-square test. Pollsters often use a T-test. These may sound confusing, but just think of them as tools that help us look at differences. The decision for which test you can run has to do with the kind of data you start with, how it's collected and whether it represents a universe or a sample.\n\nCategories or numbers?\n\nLet's talk about the different types of data. Statisticians classify variables into four measurement scales: nominal, ordinal, ratio and interval. But we often deal with just three. As you move up the data ladder, you get more power to describe an event that happened in the real world. Nominal data are used to classify things. Race or gender for example. They are categories that do not run on a scale. Ordinal data are in categories that run on a scale. Think of them as ranks. Pure ranks, like No. 1 and No. 2, are ordinal. So are words like poor, middle-income and rich. They're easy to think about. You get more information than just categories, but you don't know anything about the distance between the values. Rich may not be twice as rich as poor.\n\nContinuous variables\n\nThese are numbers: Test scores, incomes, or years in prison. Most commonly, we use numbers that can be added, divided and multiplied with ease: Five is half as big as 10. You can always convert a number into a word. You can't convert a word to a number.\n\nChi-Square\n\nChi-squares, the most common significance test used in journalism, tells us whether data arranged in categories reflect differences that happened by dumb luck. But keep in mind that we rarely deal with samples. Some would argue there's no point to a chi-square test. If the sheriff disciplined more women than men, it happened. It's true. On to the interview.\n\nBut we still have no way to evaluate the sheriff's claim of a fluke. The nice aspect of chi-square tests is that, unlike many other statistical tests, it makes no assumptions about the sample you drew. So choosing one year because it was the most recent can still create a reasonable chi-square test.\n\nYou normally get chi-square tests by setting up a crosstab, most frequently using SPSS. Although it's possible to generate them in Excel, it's much harder, because you have to hand-calculate much of its input.\n\nBefore we go on, we should probably talk a little about what makes something significant. Social scientists often say you have a \"significant\" gap if it's unlikely - about 5 percent of the time - you would have seen answers like yours purely by coincidence. It has nothing to do with whether or not the difference is meaningful, or even true. It's just by chance. The other thing to keep in mind is that significance tests rule out coincidence as a possible explanation. They don't prove anything else.\n\nAlso keep in mind that a chi-square is sensitive to two things: The number of cases you're examining, and the difference between the numbers you get and the numbers that you'd expect based on their proportions. As soon as you start looking at thousands or even hundreds of cases, chi-squares begin to report \"significant\" results, even if the difference is teeny. So a denial rate for mortgages of 7 percent white homeowners and 8 percent for black homeowners can be significant if you have enough applications. It might not stand up to the \"who cares?\" test we use for news.\n\nT-tests\n\nThe T-test is used to examine the difference between two means. They're used when you're working with one variable, or field, with numbers in it and another that comes in two categories.\n\nSarah Cohen is always looking for ideas for future statistics columns. Has a statistic got you stumped? Have you finally mastered a statistical test? Got an idea for a future story? E-mail Sarah at sarah@nicar.org",
  "provenance": {
    "extracted_by": "anthropic/claude-sonnet-4-6",
    "extracted_at": "2026-07-05",
    "verified": true
  },
  "similar": [
    {
      "slug": "1997-november-playing-with-statistics",
      "score": 0.209
    },
    {
      "slug": "1996-april-it-ought-to-be-a-crime",
      "score": 0.143
    },
    {
      "slug": "1995-may-when-an-apple-isnt-an-orange",
      "score": 0.137
    },
    {
      "slug": "1993-march-adventures-in-sampling-using-brute-force-to-compensate-for-your-ignorance-about-statistics",
      "score": 0.127
    },
    {
      "slug": "1997-november-when-the-team-leaves-home",
      "score": 0.122
    }
  ],
  "html_url": "https://dwillis.github.io/uplink/article/1997-september-testing-for-significance.html",
  "json_url": "https://dwillis.github.io/uplink/article/1997-september-testing-for-significance.json"
}