This spring, the Detroit Free Press announced that it would no longer rank schools and districts based simply on scores from statewide tests. The newspaper reached its verdict after conducting a six-month computer analysis of results from the Michigan Educational Assessment Program. It found that poverty and other factors outside a school's control were so strongly linked to test scores that it made straight-up comparisons "inevitably flawed" and "mostly meaningless."
Instead, the Free Press vowed that from now on it will produce a more nuanced picture of how well Michigan's schools are doing, given the challenges they face. With that commitment, the Motor City daily joins a growing number of newspapers that are investing heavily in time and resources on special reports that go far beyond the mere reporting of test scores. Many now produce regional report cards on schools.
The newspapers' reports often surpass the documents produced by states and districts in their level of detail, sophistication, and accessibility. Most are available on the Web.
Applying pressure
For educators, who are already pressured by demands for greater accountability, these reports create both new challenges and opportunities.
Many teachers and administrators say they welcome the potential for a deeper, more complete picture of education. But they also worry that some of the analyses may be as misleading or incomplete as the raw rankings they replaced.
"I congratulate any newspaper that attempts to put some of this stuff into context," said Linda Leddick, the director of research, evaluation, and assessment for the Detroit public schools. "For too long, they've been running test results as if they're scores in a horse race, and that does not help the public understand."
Public demand
The media's penchant for producing report cards stems, in part, from the public's hunger for information about schools.
"Many people today shop for school districts as intensively as they shop for homes," said Neill Borowski of The Philadelphia Inquirer, which published its first report card last September. The public's appetite for information comes as newspapers have greatly expanded their ability to sift through large amounts of data thanks to the advent of personal computers, Web sites, and computer-savvy reporters and editors like Borowski. The National Institute for Computer-Assisted Reporting, which trains journalists to analyze databases, regularly focuses on school reports as part of its training sessions. "Almost every major regional paper now is making some attempt to look at school reports on their own," said Sarah Cohen, NICAR's training. "I think they consider it a basic public service." Since 1996, The Seattle Times has published an annual report on schools in its region. This year, its 256-page book includes statistics on more than 530 public and private schools. Readers can also access the information on the Web. In a few seconds, parents can determine which high schools push their students to take algebra or which schools assign at least three hours of homework a night.
Creative conditions
The Charlotte Observer's report card includes snapshots of about 500 public schools in its area, including such "top-20 lists" as which schools have the most 3rd graders reading at or above grade level and which have improved the most on the state's U.S. history test.
Last month, the Los Angeles Times published a special series on California's 8,000 public schools that combined information from dozens of databases. Among its findings: Dropout rates are down, and students from all racial and ethnic groups are taking more college-preparatory courses than in the past. But the newspaper also found more than 1,000 schools that failed to move a single student out of bilingual education last year.
Some newspapers are going beyond test scores and databases to conduct surveys and poll their readers. The Seattle Times and The Philadelphia Inquirer, for example, send questionnaires to school districts, either to collect information that the state does not or to publish it in a more timely fashion. Newspapers also combine databases to make comparisons across districts or schools, often an analysis not available elsewhere. The Los Angeles Times merged data sets from the state education department and the University of California and California State University systems. "What we needed to do was to put together something that didn't exist," said Richard O'Reilly, the newspaper's director of computer analysis. The resulting combined database "pulled together all of the data from the three sources into a single record per school per year."
A level comparing field
To measure the effects of poverty and other non-school factors on achievement, the Free Press and other newspapers use sophisticated statistical techniques like multiple regression analysis.
Such methods can determine to what extent variations in test scores are related to differences in such factors as family income, student mobility, or limited English proficiency. The findings are used to create projections of likely test results for a school or district based on its student population. Schools or districts whose actual scores are much better than predicted are judged to be particularly effective at serving their students. Based on its study, the Free Press concluded that the Detroit public schools were beating the odds, while some wealthier suburbs could be doing more. Similarly, the Omaha World-Herald last year identified 10 elementary schools that had done far better over the preceding five years than their demographics would have predicted.
Everyone's a critic
Many educators praise the attempts by newspapers to put test scores in a larger context. But newspapers face many of the same problems as education researchers: Are they controlling for the right variables? Are they using the most appropriate statistical techniques? Are they reaching premature conclusions or inferring causal relationships where none exist?
The Free Press has been criticized by some researchers for what they say is an overly simplistic analysis of the non-school factors that affect test scores. Shawn M. Quilter, an assistant professor of education at Eastern Michigan University, said the newspaper's report "has a huge impact on administrators and teachers." Though it is just an analysis, he added, educators "take it as authority and fact."
Journalists counter that too many states and districts shy away from painful comparisons of schools and districts. Or they conduct such analyses and fail to publish the results. For years, the World-Herald fought with Omaha school officials over the district's refusal to provide test scores in a way that the paper could analyze them. Once the newspaper obtained the data, "we learned that the district does the precise type of analysis that we were doing to watch its schools but never made that public," said staff writer Carol Napolitano.
The Sun in Baltimore publishes a report card that profiles elementary schools in its metropolitan area. While the information is on the public record, said Mike Himowitz, the paper's electronic news editor, "the state has always seen fit to publish this data in a way that doesn't make it easy to compare one school with another, which is why our reports are so popular." Other journalists praise the cooperation of state education officials. "The state people who maintain all the data were actually thrilled that somebody was interested in using it," said Bill Ristow, the education editor for The Seattle Times.
Invest wisely
But educators and journalists alike warn that newspapers embarking on computer analyses must invest the time and money to get it right. Many of the journalists interviewed for this story had spent six months to a year on such projects and had hired expert consultants to look over their shoulders or conduct some of their analyses.
Heather Newman, the CAR specialist for the Free Press, agreed. "I think that this is a real tricky area for newspapers to get into," she said. "For people who haven't had an adequate education in the proper use of statistics, it's really easy to come up with a bunch of numbers and then to make some meaning out of them."
Numerical starting points
One of the strengths of newspapers is their ability to supplement data with more traditional reporting. "I don't think the numbers can be the central story," Napolitano said.
For its report, the World-Herald sent five reporters into a dozen schools for two weeks to visit classrooms and interview parents, students, and educators. "The more we learned about test scores and learned their limitations, the more we felt the need to get into classrooms," said Mike Reilly, the newspaper's projects editor.
The Philadelphia Inquirer incorporates "points of pride" into its report cards, in which districts identify things they are doing that stand out. "As much as we love the numbers and love the data analysis," Borowski said, "there's a lot of things you can't capture with the numbers."