Using regression analysis
When Carol Napolitano and Paul Goodsell wanted to study the effectiveness of Omaha schools, they turned to a tool that reporters increasingly use: regression analysis.
The method is growing in popularity among education reporters, as NICAR found in our first-ever seminar on advanced computer-assisted reporting techniques in Philadelphia last month, which focused on reporting on education statistics.
But the Omaha World-Herald's experience shows why using multiple regression analysis a technique that's become a buzzword in newsrooms-require more than some sophisticated software and a desire to show it off.
Napolitano and Goodsell spent weeks inside the schools they identified long after their statistical procedures were completed. What they saw not only put faces on their results, it explained them. In underperforming schools, Napolitano saw chaos. In exceptional schools, she saw teachers who returned year after year, discipline that kept disruptions at bay and strong principals.
Defining Reality
First, a few definitions. What is regression analysis? It's just a tool that helps you level the playing field by predicting one number based on values of other numbers. In Omaha and other places, that meant predicting test scores based on demographics of the student population, such as poverty, English language skills and stability (called "mobility" in school reports).
Typically, reporters subtract the predicted scores from the actual scores. What's left over is what U.S. News & World Report has started calling "value added" in its annual evaluation of American colleges and universities. It's the improvement that's attributed to the school's effort rather than the advantage or disadvantage that students bring to the schools from home.
The results can be striking. In Omaha, almost half of the schools that performed worse than the analysis would have predicted on California Achievement Tests scored better than the median county-wide. Almost half that outperformed the prediction scored below the district median. Hanging out at the schools confirmed their statistical results. At the most effective schools, teachers kept order, gained the respect of the kids and challenged them to work hard. At the least effective, kids slept at their desks, flashed gang signs and wandered around aimlessly with a helpless teacher standing by.
Lessons Learned
Omaha's statistical technique can become a lesson for many reporters hoping to make sense of the proliferating school reports. The first battle was getting good data. What Omaha learned was that more isn't necessarily better-just more flexible.
After years of fighting, Omaha won access to a database containing 70,000 individual student records on the California Achievement Test, the home ZIP code, the school and a couple of other identifiers including participation in school lunch programs. What it meant was that Omaha, unlike others who have tackled this project, had actual scores rather than the percentage of kids who scored in certain ranges. It also meant they could match the ZIP codes with demographic information about the home neighborhood, not the school neighborhood, in a district with busing.
What they ended up using in their analysis, though, was familiar: All but one was a traditional measure of poverty or advantage, ranging from the percentage of parents in the home ZIP code with college degrees to the percent of students on free or reduced lunch. The last was the percentage of minorities taking the tests. Together, these factors accounted for up to 84 percent of the difference in median test scores among Omaha schools. The other 16 percent was, well, news.
If Omaha wanted to predict school performance, why did it exclude information on, say, dollars spent or PTA hours logged? Its goals, unlike those of the Arkansas Democrat-Gazette featured last month in this space, were to isolate the characteristics of a school that teachers, principals, or officials, couldn't control. That way, it could evaluate the extra value each school brought to education rather than predict more exactly the school's performance on a series of test measures.
The method wasn't without controversy. School officials told Omaha reporters and editors that they'd ruin the lives of each 2nd grader in the district. They condemned the method. A school official's daughter, who was a graduate student nearby, e-mailed daily to question another detail of the method after the series ran. But the analysis had one strength the school officials couldn't deny: It confirmed internal, unpublished studies conducted by the school system itself in its attempts identify the best and worst of Omaha's schools.
And it did so with compelling anecdotes from inside those schools. Here are some tips for reporters who want to tackle something like Omaha's project:
Fight for information on students rather than settling for the school reports offered up on the Web. This information was crucial to Omaha's ability to look at a range of student demographics using Census data.
Identify your goals early on. In Omaha, reporters wanted to evaluate the performance of the education system in each school. In Arkansas, reporters decided that it didn't matter why schools were good - they wanted to develop a model of quality that ignored socioeconomic factors. Both are legitimate goals. Just decide which is yours.
Watch for bad data, but don't get paralyzed. In Omaha, for example, certain schools were designated "universal free lunch" schools. So many of the kids qualified that the schools just gave the program to everyone and stopped counting how many would have gotten the reduced price lunches elsewhere. They didn't show up as poor in the database.
Look at more than one year's worth of information. Omaha looked at schools that under- or over-performed five years in a row.
Use the scientific method, and understand all of your results.
If you need help, hire it. Omaha, in fact, used two consultants, John Bare and Brenda Sugrue, to lead them through the maze of numbers and tests.
Don't let the school officials intimidate you. They'll always tell you that you've made big mistakes, and you're irresponsible. If you have been responsible, don't let it dissuade you.
Sarah Cohen can be reached at (301) 942-2199 or e-mail her at sarah@nicar.org