Few things are as alarming as analyzing numbers for the second or third or fourth time to check on them before publication only to find you're getting different answers. This probably has happened to all of us, regardless of the story or computer program. For years, it has happened with regularity when I compute percentiles.
The scenario usually goes something like this: I set out to find percentile "breaks" for school test scores. My first run is in SPSS. In Frequencies, you can select the number of "cut points" (for example, five cut points yield the 20th, 40th, 60th, 80th percentiles). Or, you can specify each percentile in SPSS. My second run is in Excel. I use the percentile function and more often than not come up with answers different from those found with SPSS.
Avoid believing that the software must be correct. The evidence shows there is no cause for blind faith.
After overcoming the churning feeling in my stomach, I check the breaks and find they cut the data into the same groups. In one simulation, SPSS set the 20th percentile break at 95.5 while Excel set it at 98.6. A visual inspection of the data shows the break comes after 94.5 and before 99.7.
Different algorithms? Probably. I'm still waiting to hear from the software companies. The point is: The answers were different. Could there be other conflicts between programs that aren't as readily apparent?
Should we, as journalists and analysts, worry?
If we worry, we won't be alone.
Economists B.D. McCullough of the Federal Communications Commission and H.D. Vinod of Fordham University sounded the alarm in the June issue of the Journal of Economic Literature, the widely respected and refereed scholarly journal of the American Economic Association.
Their 30-page article details the authors' benchmarking of econometric software packages and similar work in the past by other econometricians. "Apart from cost considerations," the authors note, "economists generally choose their software by its user-friendliness or for specialized features. They rarely worry whether the answer provided by the software is correct."
The authors surveyed five journals that published more than 120 reviews of econometric software between 1990 and 1997. "All but three paid no attention to numerical accuracy, and only two applied more than a single test of numerical accuracy," McCullough and Vinod found.
They tested four econometric software packages and found varying answers. (They didn't identify which packages, saying that was the job of software reviewers.) Their tests and descriptions of others were complex and economics-heavy. In one test, only one of four programs correctly calculated the correlation coefficients in a problem. Only two of the four correctly calculated the standard deviation in the same problem.
Recall that a correlation coefficient of 1 is "perfect" correlation variable moves in lockstep with the X variable. One software package tested by the authors found correlation coefficients greater than 1 (1.127 and 1.137).
"Many textbooks convey the impression that all one has to do is use a computer to solve the problem, the implicit and unwarranted assumptions being that the computer's solution is accurate and that one software package is as good as any other," the authors wrote.
The authors call for more rigorous testing of software. They call on scholarly journal editors to require authors to identify their software, including the version number. And they call on other economists to demand accurate software rather than shopping only for "speed, user-friendliness and the latest econometric features."
A fundamental tenet of research is replicability - the ability of researchers to come up with the same results of other researchers given the same methodology and data. Software accuracy problems undermine this basic truth, the authors contend. "An unfortunate consequence of two different packages providing two different answers to the same problem is a lack of replicability in economic research," they wrote.
Now, back to journalism and our brand of data analysis.
If we don't worry, we should at least be aware. Test your findings with at least two different computer programs. When possible, calculate one of the statistics by hand and compare your results with the software's.
And avoid believing that the software must be correct. The evidence shows there is no cause for blind faith.
Neill Borowski's can be reached by e-mail at inborowski@phillynews.com.