Turning print-image files into databases

Journalists can convert messy print-image files from government agencies into usable databases by using word processing programs or specialized software to automatically remove headers and formatting rather than manually deleting them.

In the land of data garbage one of the most painful files to get from a government agency is a print file.

This file looks just like the information you get on a printout and contains header information that looks like "page one, judicial files of america, section 203AB, the office of Eddie Haskell, 10/20/92" and so forth for each "page" of information.

If you can't convince the agency to give you a nice, flat file then it's laundry time.

You can clean these files, if they are small, by throwing them into a word processing program like XYWrite III Plus and manually defining the offending headers and deleting them.

This won't work very effectively for a 10 megabyte file, however, and that is where, whether you are a programmer or not, you should head for a program.

At The Courant, I discovered that a good person in our publishing systems had a handy XYWrite III Plus program that with a little modification would go in and destroy headers until it reached the bottom of the file.

All you have to do is identify a mark or string of characters that establishes the beginning of the header - "page" for example - and the end of the header - "92" perhaps and run it through its loops. Sometimes you might have to make other pass through the data, but the job gets done. I know the same kind of program can be written more gracefully in other word-processing programs or database programs, but the message of this tip is not to waste time on manual deletion when a little programming can help. Also, I have been told that a database program - Monarch - takes care of this problem during importing and look forward to reading about it while trying to find money to buy it.