Sometimes the numbers look just too good to be true. That's when it pays to be suspicious. Two days after the mysterious crash of USAir Flight 427 near Pittsburgh, Elliot Jaspin and I were looking for a common thread in the Federal Aviation Administration's service difficulty reports on 737s. Elliot had obtained a five-year file of SDRs by e-mail from the National Institute for Computer-Assisted Reporting. Unzipped, it took up more than 30 megabytes. Using FoxPro, he sorted the records by level of severity. The FAA ranks its SDRs on a scale of 1 to 5, with 1 having a "seldom relationship" to an accident and 5 having "frequent relationship" to an accident. We decided to just look at the level 5 reports. He then sorted the level 5s by the type of problem, looking for some pattern of difficulty with the 737s. By far, the biggest area of difficulty was "Other." We looked at all the other areas, but no significant pattern emerged. We looked closer at "Other," and, again, no pattern emerged. Then Elliot sorted the data by airline and by unique plane numbers. Suddenly, it appeared that we had a pattern - and a helluva story. Dividing the number of SDR 5s for each airline into the number 737s in its fleet over the past five years gave us a percentage. From this calculation, it appeared that two-thirds of USAir's planes had reported a level 5 SDR. That was among the highest percentage of the major airlines. It looked like we'd made a significant finding. I called USAir for their reaction to our hypothesis that two-thirds of their planes had experienced major service difficulties. They said they weren't sure what the data showed and defended their record as a safe airline.
We decided to run the data past aviation safety experts. I posted an e-mail query on Profnet. Within a few hours I had the numbers of several aviation safety experts affiliated with universities, and they led me to others in the field.
After a day of talking to experts, I was convinced our hypothesis was wrong. The numbers were accurate; they were also misleading. Every expert I spoke with said that publishing a story along the lines I suggested would be unfair. They gave two main reasons:
- Not all airlines report SDRs consistently. At least one expert noted that USAir, which has a relatively young fleet of 737s, was reporting SDRs twice the rate of two airlines with substantially older 737s. That might indicate that USAir was simply more scrupulous about reporting its SDRs than the other airlines.
- While the number of SDRs compared with the number of planes gave us a percentage, it was meaningless. When you looked at the thousands of times each airline's planes had flown over the past five years, the percentage of SDRs to flights was minuscule. To say that two-thirds of the planes had problems, when far less than 1 percent of the flights had problems, would have been misleading and sensationalist.
Faced with such comments by aviation experts not affiliated with any airline, we decided to scrap the story. But we still had something; we had information that a data system used by the FAA wasn't very reliable or meaningful. We decided to find out what the FAA did with this data and here, ultimately, is where we found our story. It turns out that the FAA admits the data from the airlines is neither consistent nor used as well as it might be for statistical analysis of either the airline's performance or safety trends. Virtually all the analysis is done manually because the FAA doesn't have the staff or the computer resources to better handle the data. That's the story we ran. - Larry Lipman is the Washington correspondent for The Palm Beach Post in the Cox Newspapers Washington Bureau. Elliot Jaspin is Systems Editor for Cox Newspapers Washington Bureau. They can be reached by e-mail at jaspin@access.digex.net.