A look at how Congress spends its money was USA Today's first experience with scanning. Perhaps it was the size of the project (7,500 pages) or the number of records (325,000) or the complexity of the subject - whatever, it turned out to be a nightmare.
USA Today and Gannett News Service wanted to look at what it costs to run Congress and to identify, among other things, the top spenders. We started by asking Congressional leaders for a computer tape of all 1991 expenses by House members' offices, committees and support departments. They refused, saying all the information was publicly available in four books kept by the Clerk of the House. At least one person acknowledged that providing the information on tape, rather than paper, would make analysis easier.
In order to proceed with the project, we chose to scan the four books into the computer. The plan was to have the tabular data scanned and converted into fixed-position ASCII files, which we could then read into Paradox.
We quickly dismissed buying a desktop scanner and doing the work ourselves because of the volume, and the slowness and inferiority of portable models. Instead, we hired a company that estimated scanning on its professional machine would take four weeks. Try four months.
The company used the Xerox Imaging Systems K5200, a Kurzweil hardware and software system priced at $14,500. Stacks of pages are automatically fed into the scanner, much like a high-volume copying machine.
This is a very time-consuming process. The book's agate type stymied the splitter. The splitter software is designed to separate touching characters, making it possible to "read" them. If it splits improperly, all sorts of strange things can happen. For example, the letter O could be read as a C. The splitter has problems with small and bold text and with characters that are angled or italic. Well, the House books are all of that.
Any stray or unwanted marks on a page, such as ink smudges and page numbers, introduced errors.
Among our problems:
On many pages, the Government Printing Office had not printed the type square on the page. The software has a tolerance of about 5 percent skew. anything more than that requires an operator to manually place the page in the machine.
The result: Lines of garbled or wrong characters: 7's appeared as I's, 8's as 3's. (Some scanning errors were comical - shamed employee, instead of shared; resistant executive, instead of assistant.) Data from two lines often converged into one. Plus, the tabular structure of the original data was often lost, creating errors later when we imported the fixed file into Paradox.
It took months of extensive work to clean up the ASCII files. It was crucial that every number be double-checked with the original. And, in order to do any collapsing of data and analysis, we needed to correct any errors in staff names, job titles and government contractors so they were consistent.
Although it's hard to estimate the overall error rate caused by scanning, on some pages more than a fourth of the characters or numbers were wrong. Before you proceed with a scanning project carefully consider the quality of your originals. Is the type clear, preferably not agate, smudge-free, and printed straight on pages? Testing is crucial. You should have several dozen, representative pages scanned and then doublecheck the electronic data with the original. How many errors were introduced and how difficult will correction be? Is scanning worth it if the cleanup is extensive?
In researching this article, our contractor told me that scanning technology has improved dramatically, even since we began our project a year ago. We hope so. The quality of the conversion we got was so poor that, had we known at the time, we wouldn't have gone the scanning route. Instead we'd have cut back on the data we wanted from the House books and had it double-keyed.