Playing by the numbers

Playing by the numbers

By Andy Lehren, NICAR staff

The holy grail in aviation data is figuring out what predicts accidents.

What causes plane crashes: The airline? The type of aircraft? Pilot mistakes? Poor maintenance? Violent weather? How can reporters sift through patterns when so much changes over time - airlines change management, the aircraft they fly, and even owners.

After the ValuJet crash, a challenge remains in trying to develop an airline safety index. And even if this is not successful, University of North Carolina Professor Phil Meyer, author of "The New Precision Journalism," said this problem sheds light on how to think about developing an index.

Indices are no stranger to news pages. Look at the Dow Jones Industrial Average or the Standard & Poor's 500. Both are used daily to reflect the stock market's health. The Philadelphia Inquirer's Neill Borowski wrote in April's Uplink how he indexed various local economic segments against the nation. That helped him determine the Philadelphia area's strengths and weaknesses. Baseball fans track how many games back a team is from the leaders.

More than ranking

The idea is that these numbers do more than rank things. They are barometers. They show relative strength or weakness. A baseball fan knows more than whether a team is in second place; games back tell whether the team is close or far from the division leader. The S&P 500 indicates how much better or worse the stock market is doing. The index used by Borowski showed how much the local economy's makeup is different.

At Newsday, Ford Fessenden had previously looked at accidents worldwide to show a history of airline crashes, including predecessors in the numbers for the airlines that took them over, and included information such as the odds of a plane crashing. That, he said, showed readers that foreign carriers in certain countries have troubling accident rates.

The stories also underscored how human mistakes - and companies with a culture that did not emphasize safety and training play a great role in crashes.

Meyer suggested reporters can try to go one step further. The goal would be to create a safety index for airlines, gauging the likelihood that someone would get killed riding a particular airline. It would include an analysis looking at whether factors in older crashes predict more recent accidents.

Drawing from the data

An index would use one or more factors for measuring safety. Journalists look to Federal Aviation Administration data such as Service Difficulty Reports, which include equipment problems, engine flame outs and emergency landings. They also look at the FAA's Accidents and Incidents reports, which include when people are hurt or killed, or the airplane suffers major damage.

The FAA also keeps information on accident rates, and one recent report, separating start-up carriers from major airlines, claimed there is a difference between the two groups. The report ranks airlines by accident rates, but does not include a statistical analysis.

Reporters have already documented airlines failing to report SDRs. Instead of representing the whole universe of U.S. airline problems, or even a random sample, the database is skewed because of contrasting reporting rates. Airlines are required to file SDRs when they repair aircraft, but reporters have found many don't comply with FAA rules, and the agency lacks the teeth for enforcing regulations.

Richard Newman of U.S. News & World Report is among those reporters who believe SDRs could never be part of an index on airline safety. In an unpublished analysis, St. Paul Pioneer Press' Dan Browning attempted to adjust SDR data by looking at reporting rates. The Cleveland Plain Dealer reported nine of 10 repairs were unreported in SDRs (See Uplink, February 1995). Even when reports are made, key information may be missing. Details may not be coded consistently. Accidents and Incidents data also suffer from missing information, though many reporters agree that accident data appears more complete.

What if a reporter did not face these and other problems, Meyer said they could begin the quest by looking at whether fatal air crashes and major repairs during the 1980s predict accidents during this decade. "It may be that the past doesn't predict the future," Meyer said. "If it doesn't, there is no reason to look at an index." But if it did, reporters could also look at how well several factors such as aircraft age and the number of most severe SDRs - fit together. One test for reliability is called Chronbach's alpha, which looks at whether factors move together consistently. (In SPSS 6.1, with professional statistics, go to statistics scale reliability analysis). Those who attended the 1996 Advanced Computer-Assisted Reporting Bootcamp in North Carolina used it for studying polling data. The most robust indices, Meyer said, are those that still predict even when some elements are stripped away.

Because the FAA's inspector general raised so many concerns about ValuJet before the crash, Meyer said the quest for developing a safety index is worth pursuing. But one problem - for data hounds, not passengers - is the few number of crashes in U.S. aviation. That means an analysis trying to predict accidents is fragile. To get an idea of what that means, if you changed an airline's accident total just by one crash, it could drastically alter results. The FAA noted that in its recent report on start-up carriers. One way to shore up that problem is to also look at severe mishaps and other incidents. But those records - along with those on pilot mistakes and other human errors - are less complete. A recent New York Times story noted these problems for developing a safety predictor. Trying to use airline data to do this kind of analysis, said the Minneapolis Star Tribune's Greg Gordon, "is a lot like squeezing molasses and watching it go in all directions."

Andy Lehren can be reached at (573) 882-0684, or send e-mail to andy@nicar.org