One stumbling block that reporters face when they begin working with statistics is the timeless cause-and-effect debate.
When virtually every book on numeracy and statistics lectures us about our constant confusion between correlation and causation, it may seem wise to avoid any analysis that will thrust you into the debate.
But complaints are often a little hysterical. I doubt many of us would write stories ascribing (a) more drinking to more clergymen in town, or (b) higher stock prices to sun spot activity.
Even so, the point is well taken. It's all very nice to say that the failure to get a mortgage or the amount of bail set are associated with a person's race. Those only become stories when we can say why. We aren't interested in just linking two events; we want to show why they're related.
Statistical tests such as chi-squares and regression results can tell you whether a relationship between two events could have happened by dumb luck. But they can't tell us whether one event caused another.
More bad news: We rarely conduct controlled experiments, the only kind of research that can tell you cause and effect. Instead, we work from a mass of public records that were collected for some other purpose. These records often miss the key elements and defy our attempts to control for even simple and obvious confounding events.
School report for instance, lack the single biggest indicator of poor performance - a definitive measure of poverty. Home mortgage records lack any information on credit records. And health records lack medical histories.
Validating research
At least one author is ready to help us through the steps that others take when they want to say that one event may cause another. In Statistics: A Spectator Sport, author Richard M. Jaeger reveals the ways researchers decide whether they've found a cause and effect.
We usually only care about what researchers call "internal validity" of our work. External validity would mean that we care about generalizing from our specific work to a bigger world - something we rarely do. So here are some of the guidelines:
Does the cause come before the effect? This seems obvious, but it isn't always. Test scores may depend on the number of teachers in a classroom. The timing, though, is relative. Spending may, in turn, depend on performance just last year. Performance in a school doesn't usually change that quickly.
Is there a third event that causes the relationship? Again, a school example is worth noting. Most reporters would find that high minority populations in schools are associated with low test scores. But a third factor - poverty and education of the parents - plays a role in both the cause and effect in this case. Although ethnicity comes before just about anything else, few (outside The Bell Curve crowd) would argue that it causes any of the difficulties minorities face in school.
Is there another explanation? Can you think - or have your sources thought up - of rival theories that are equally sound? Look to your reporting effort rather than statistics for this step.
Does anything refute the theory? Are there other studies that fail to show the expected causal relationship? Why might your theory only work locally? Why might it fall apart in another year?
This is an uncomfortable position for us. It involves an about face in our normal reporting process - faced with disproving our own theories rather than proving a tip. And it leaves us open to criticism we've never faced when collecting quotes or uncovering documents.
But there is good news. One way to figure out cause is to watch and listen for anecdotes that are described by the numbers.
Now that's something we're happy to do. In fact, we prefer it. We'll search for weeks to find the people who are described by our statistics. It's there - not in the numbers - that we find cause. Statistics can document a pattern and tell us how consistent it is. Reporting tells us why it happens.