Ancient warrior wisdom is based on the belief that the battle is won or lost before the first sword is drawn. Likewise, the fight for data must be plotted before the first request is filed. Understanding the buzz words the warrior language, if you will - will go far in negotiations.
Study the file layout
Skip this vital first step at your peril. Trust me. Take time to do this or risk major setbacks in time and money. The file layout is a document that shows data organization and structure. It serves as a blueprint for obtaining, importing and analyzing data. Codekeys are a list of standardized abbreviations, acronyms or numbers that represent information. For example, states may be represented as numbers in the database.
Study file layouts before making requests. For example, the database layout of the National Highway Traffic Safety Administration lists hundreds of fields involving fatal vehicle crashes. One field, called the "related factors" field, uses a code number to designate if the fatality was related to a police pursuit. An entire series can and has been built on one field.
Layouts also reveal a major pitfall for the unwary: packed fields.
Typically, packed fields hold information that is encrypted and condensed as a space-saving technique. Unless fields are unpacked, you may not be able to read the information.
Often, government data is stored in multiple files, called a relational database. Examine layouts for a common field, which is necessary for linking files.
Handling Formats
This is the step where some bureaucrats love to leave reporters in the dust. Learn about the types of formats your software can handle. However, most any database can be copied as an ASCII file. This is an acronym for standardized computer format that is recognizable by all mainstream software (FoxPro, Access, Excel, FileMaker Pro, etc.)
Most software also can handle multiple formats, such as DBF or text format or formats managed by software applications.
Next, you must determine how the data is delimited - how it will be separated. Many reporters will ask for data in ASCII format (an industry standard) and delimited by commas. During importing, the computer will separate the data into new fields at each comma. You can have the data delimited by spaces, or tabs, or most anything, if preferred.
Medium messengers
You've followed the first two steps flawlessly, the agency quickly delivers your data - and it's on a 4mm tape cartridge. This happened at the Tribune when a reporter forgot to request the medium, which is the all-important third step. Needless to say, few newspapers have 4mm tape drives to read the data. Determine what formats your paper can handle, such as disks or cartridges or tapes. Besides disks, here are some other choices:
Nine-track tapes - Used by many government agencies, but you will need a special nine-track reader attached to your computer. JAZ Cartridges - These tapes hold 1 gig or more of data. JAZ drives are common and increasingly more inexpensive. ZIP Cartridges - Similar to JAZ, but typically these cartridges hold less information. FTP (File Transfer Protocol) - This a storage medium, but it's a dream way to obtain data. In short, the agency temporarily places the requested data on its Web page, allowing you to transfer the data via the Web into your computer. No tapes, no disks, no special readers. And it's more efficient for agencies.
Roadblocks and solutions
The pursuit of data is no laughing matter, but government response can border on the ridiculous. Here are actual government objections encountered nationwide. The successful strategy centers on getting past the flak phalanx and talking directly to the agency's data processing employees.
Our computer can't do that. You'll never understand the data. It's far too complex. We're afraid you'll make mistakes and draw wrong conclusions. It's too big. You'll never have a computer to handle it. Our software is proprietary, so you can't have the data.
Data negotiations are fought with knowledge and confidence. More than once I've had a new term thrown at me during a meeting. Don't panic, don't flinch. Keep smiling, say no problem, then later run to someone in the newspaper's data processing department to find out what the term meant - and whether it's actually going to be a problem. Don't let the agency bluff you, but don't be afraid to do a little bluffing, too.
Demand specifics, then demand proof. IRE and NICAR have excellent resources for those confused by techno-babble such as megs and gigs (terms related to data storage).
Don't be afraid to ask basic questions. Ask the agency computer people how they relate multiple files: you may learn their method is more complex - but more accurate - than your plan.
Another psychological negotiating ace is to assure the agency that you plan to provide them a copy of all analysis (we're talking aggregate conclusions and totals) prior to publication. I love the moment I get to show the agency what their own data reveals. And more than once, I've been saved from publishing incomplete conclusions. Remember, the potential for errors is exponentially higher in CAR reporting. You're just one keystroke away from an error, always.
Michael Berens can be reached at (312) 222-4798 or by e-mail at mberens@tribune.com