Sampling for answers
It seemed easy. Go to the state capitol in Baton Rouge for two or three months and see how well the Louisiana Department of Environmental Quality is protecting the earth, air and water.
The agency has long been a lightning rod for controversy, some of its own doing and some because of the sensitivity of its duties. Of particular interest was a 1998 study by the Tulane Environmental Law Clinic suggesting the DEQ was rarely penalizing polluters for environmental violations. The study, however, raised as many questions as it answered.
Louisiana's three New York Times Regional newspapers - The Courier, the Daily Comet in Thibodaux and the Daily World in Opelousas - decided to look closer. It was not an easy decision; the three newspapers have fewer than 25,000 readers each. But the editors decided the story was important, so we pooled our resources and tried to tell the story as best we could. I was picked to lead the investigation.
The difficulty in finding clear answers to simple questions, however, quickly turned three months into seven.
In the end, the resulting five-day series, published Feb. 21-25, showed that the state's top environmental agencies failed on many occasions to resolve long-standing pollution cases and hold the wrongdoers accountable. It also showed that the agency lacked the kind of critical information that could allow ordinary citizens to make their own assessment of the agency and how well it protects the state's environment.
Data surprises
From the beginning, the plan was to obtain DEQ's electronic data on enforcement and let that tell the story.
That was easier said than done. As it turns out, DEQ has lots of data. But little of it covers more than any one branch of the agency, and most of it is full of holes and riddled with errors. In fact, the agency had more than 250 databases from which to choose. We eventually submitted public records requests for about half a dozen databases.
To help us extract the truth from this mess, upon the advice of a couple of IRE members, we sought outside help. I had had some limited experience dabbling with spreadsheets, but I felt I needed someone with more expertise to come along for the ride. Unfortunately, Louisiana is not blessed with many journalists familiar with databases and other computer-assisted reporting techniques. So, we placed want ads in a couple of newspapers. We ended up reaching outside the profession and hired Ashley McManus, a computer programmer living in Baton Rouge.
For close to two months, Ashley and I wrestled with this information, but were largely defeated by its many inherent flaws. The data had no common denominator. Typos were common. Many seemingly important data fields were started, but often unfilled or contained infrequent entries. In fact, only a few fields were even close to complete.
It was a classic "dirty data" situation. We tried to compare the numbers with official totals, but we found to our surprise that the agency collected no reliable official totals of its own which we could use to check our results. We were able to use this information only sparingly.
To get a more complete picture, I sought advice from Steve Doig, a professor at Arizona State University, an IRE member, and an unofficial adviser to this project. He recalled a crime study he had performed at The Miami Herald years ago in the days before desktop computers. Instead of obtaining a database, he pulled an old-fashioned statistical sample of a couple of hundred paper files. It was not of the caliber of a Gallup Poll, but the sample was helpful for discovering overwhelming trends.
Sampling
We decided to pull a random sample of our own. It consisted of 200 DEQ cases selected at random from more than 11,000 cases contained in the agency's main enforcement database. On two laptop computers, we created Excel spreadsheets and filled the headings with an ambitious list of items we wanted to check. And then for another month, we visited DEQ public file rooms and pored over documents.
But once again, poor record-keeping caused problems. We were unable to locate a third of the cases. So one of the stories became the poor quality of the agency's data. We were also able to show that a significant proportion of the cases were either still open or had a status that could not be determined. This confirmed the impressions of many citizens that the agency was slow to act decisively and would tolerate pollution for years without bringing the cases to a conclusion.
The search also yielded specific stories. Three of the cases uncovered had local significance to the readers of our three newspapers. They were cases of pollution which DEQ proved ineffective at stopping. Reporters at each of the papers then pursued these stories in greater depth.
One warning
Be careful about borderline trends. Our error ratio was 6.6 percent, which would be too high for a professional statistician, though it was fine for our purposes. If we had declared that a slim majority, or 54 percent, of the case files were missing, we would have been wrong. The real answer would have been that between 47.4 and 60.6 percent of the cases were missing. That's probably a majority, but not necessarily. We would have handed our critics ammunition.
Obviously the sample wasn't the end of our investigation. We still had a lot of work to do, interviewing sources and amassing various government documents. We also managed to use some of the more recent DEQ electronic data for some limited comparisons.
The sample, though, was crucial. Besides the authority it lent to our work, it also provided a worthwhile framework with which to examine paper files. Those days reading case files proved the most illuminating of the investigation.
It's ironic that an old-fashioned sample was more fruitful than modern-day database crunching. Obviously, I would have preferred to have the data gift-wrapped for me to analyze, but with poorly run government agencies that often doesn't happen. So, if you have the time, or even if you're just fitting in this kind of work while doing other things, a study of the sort we undertook can pack a lot of punch and shine a light on worthy information that you would not have otherwise obtained.
Charles Lussier can be reached by e-mail at charles.lussier@houmatoday.com