You might win public service awards for studies of giant government databases on highway contracts, creative campaign financing or the redlining practices of mortgage lenders, but to twist an old saw, the way to your readers' hearts is through their bellies. Everybody eats. The health departments in many metropolitan areas have computerized their restaurant inspections, opening a larder of data to news organizations with the nerve to peek inside the kitchen - or under the dumpster.
Last spring, Jan Paul, a projects editor at the St. Louis Post-Dispatch, assigned three reporters to examine restaurant inspections after the public became concerned about increasing reports of E. coli bacteria poisonings in the Pacific Northwest and hepatitis outbreaks in St. Louis. In March, the newspaper got computerized data from St. Louis County and the city of St. Louis listing 103,493 inspection reports on 8,544 restaurants, delis, snack bars, institutions, corporate cafeterias, mobile kitchens, schools, nursing homes, daycare centers and retail bakeries.
(St. Louis County later updated the records, adding 8,524 inspections to the mix.) Restaurants that get Bs or Cs must correct their problems within two days or they can be shut down. Similarly, restaurants that get hit with critical violations must correct them within a couple of days - but the public might never know there was a problem, because a restaurant could have up to three critical violations and still get to post the "A" rating in its window.
In most places that use the FDA's model inspection form, restaurants start an inspection with 100 points (a perfect score). Inspectors look for 29 categories of minor problems, which cost the restaurant either one or two points each, and 13 categories of "critical violations" - those with the greatest potential to have an adverse effect on public health - which cost the restaurant four or five points each.
When we dug into the data, it seemed that the obvious way to measure relative performance on inspections was to examine average scores. We did that, and we found that establishments got As 98 percent of the time. In fact, they scored above 90 points in 88 percent of the cases. Ranking restaurants based on their overall scores seemed futile. This analysis, based on 44,000 inspections at 3,200 eateries, found 'A' rated restaurants could still pose a risk.
We figured that we should look at the frequency of critical violations on the grounds that it might only take one incident to make someone sick. And we decided we should look at the intensity of critical violations - expressed as the average number of critical violations per inspection - on the grounds that the more of them that existed, the greater the chance that someone might get sick. In St. Louis and St.
Louis County, restaurants get letter grades to post in their windows. An "A" means the restaurant scores higher than 84 points. A "B" can go as low as 70 points, and a "C" goes as low as 50. Unfortunately, the databases we got were so badly designed that the health departments admitted they had never been able to use them. And the data was so "dirty" with entry errors that it took us about six weeks to clean it up so that we could group the restaurants.
Once we could use the data, we decided to limit our examination of the county data to only those inspections since Jan. 1, 1990, so that it would parallel the city data. We purged places that had closed or places that had changed ownership in the study period. And we limited our analysis to establishments that had been inspected at least six times so that the data would not be distorted by anomalous inspections.
We ranked all establishments based on their average inspection scores and the frequency and intensity of their critical violations. We looked at retail eateries as a class (restaurants, delis, snack bars, salad bars), schools and daycare centers, retail operation run by grocery chains, nursing homes and hospitals. We examined scores based on ethnicity. And we examined the scores of all the individual health inspectors to see whether any patterns emerged that might indicate hanky-panky.
Because of the number of ties in critical violations, the range of possible places in our ranking system was dramatically reduced. The critical violations lost their emphasis when we merged them with average inspection scores, which had a greater range of possible rankings. Thom Gross, an investigative reporter, suggested converting all three rankings to percentiles, then averaging them together. We did that and it worked.
Several public health officials around the country said our methodology seemed sound. Two reporters took a list of eateries that scored in the bottom one percent in our lists, and began verifying every single hard copy of their inspections. We found only two errors in several hundred records, and they did not affect totals. We corrected those, and decided the overall data was statistically reliable.
We came under attack from several quarters. The director of the city's health department said our study was no more valid than ranking restaurants based on their addresses, because no link has been established between foodbourne disease and restaurant inspections. But thanks to a search I did on CompuServe before we met with him, we were prepared. We directed his attention to an FDA study in Seattle-King County, which concluded that restaurants with low scores and critical violations were directly correlated to outbreaks of foodbourne disease.
The final challenge to our project came from the Missouri Restaurant Association, which argued that the inspection system itself was flawed. We agreed, but for opposite reasons. The industry association argued that any restaurant that was open was safe. Our articles showed that the inspection system failed to communicate anything to the consumer about critical violations, that inspectors often give a break to restaurant owners if they are cooperative and that the letter grade system used in St.
Louis can be misleading. When we finally published the articles, our analysis was based on about 44,000 inspections covering 3,200 eateries. Our restaurant inspection report was a hit with our readers. Callers are praising us for it. Although it can be time-consuming, it's a project that can be done virtually any place that has a health department. But beware: The project was not a hit with the advertisers.
We were told that after the project, an editorial and a cartoon ran, advertisers cancelled a special section on dining out.