{
  "id": "1993-january-read-this-before-scanning-text",
  "slug": "1993-january-read-this-before-scanning-text",
  "url": "https://dwillis.github.io/uplink/article/1993-january-read-this-before-scanning-text.html",
  "year": 1993,
  "month": "January",
  "monthNum": 1,
  "headline": "Read this before scanning text",
  "kicker": "",
  "author_name": "Barb Pearson",
  "author_title": "",
  "affiliation": "USA Today",
  "summary": "USA Today's attempt to scan 7,500 pages of Congressional spending records proved far more difficult than expected, with scanning technology struggling with small text, formatting issues, and ink marks that created numerous character recognition errors requiring months of manual correction.",
  "keywords": [
    "scanning",
    "Congress",
    "errors"
  ],
  "topics": [
    "Data & Databases",
    "Investigative Methods",
    "Technology"
  ],
  "technologies": [
    "Paradox",
    "Xerox Imaging Systems K5200",
    "Kurzweil",
    "ASCII"
  ],
  "full_text": "A look at how Congress spends its money was USA Today's first experience with scanning. Perhaps it was the size of the project (7,500 pages) or the number of records (325,000) or the complexity of the subject - whatever, it turned out to be a nightmare.\n\nUSA Today and Gannett News Service wanted to look at what it costs to run Congress and to identify, among other things, the top spenders. We started by asking Congressional leaders for a computer tape of all 1991 expenses by House members' offices, committees and support departments. They refused, saying all the information was publicly available in four books kept by the Clerk of the House. At least one person acknowledged that providing the information on tape, rather than paper, would make analysis easier.\n\nIn order to proceed with the project, we chose to scan the four books into the computer. The plan was to have the tabular data scanned and converted into fixed-position ASCII files, which we could then read into Paradox.\n\nWe quickly dismissed buying a desktop scanner and doing the work ourselves because of the volume, and the slowness and inferiority of portable models. Instead, we hired a company that estimated scanning on its professional machine would take four weeks. Try four months.\n\nThe company used the Xerox Imaging Systems K5200, a Kurzweil hardware and software system priced at $14,500. Stacks of pages are automatically fed into the scanner, much like a high-volume copying machine.\n\nThis is a very time-consuming process. The book's agate type stymied the splitter. The splitter software is designed to separate touching characters, making it possible to \"read\" them. If it splits improperly, all sorts of strange things can happen. For example, the letter O could be read as a C. The splitter has problems with small and bold text and with characters that are angled or italic. Well, the House books are all of that.\n\nAny stray or unwanted marks on a page, such as ink smudges and page numbers, introduced errors.\n\nAmong our problems:\n\nOn many pages, the Government Printing Office had not printed the type square on the page. The software has a tolerance of about 5 percent skew. anything more than that requires an operator to manually place the page in the machine.\n\nThe result: Lines of garbled or wrong characters: 7's appeared as I's, 8's as 3's. (Some scanning errors were comical - shamed employee, instead of shared; resistant executive, instead of assistant.) Data from two lines often converged into one. Plus, the tabular structure of the original data was often lost, creating errors later when we imported the fixed file into Paradox.\n\nIt took months of extensive work to clean up the ASCII files. It was crucial that every number be double-checked with the original. And, in order to do any collapsing of data and analysis, we needed to correct any errors in staff names, job titles and government contractors so they were consistent.\n\nAlthough it's hard to estimate the overall error rate caused by scanning, on some pages more than a fourth of the characters or numbers were wrong. Before you proceed with a scanning project carefully consider the quality of your originals. Is the type clear, preferably not agate, smudge-free, and printed straight on pages? Testing is crucial. You should have several dozen, representative pages scanned and then doublecheck the electronic data with the original. How many errors were introduced and how difficult will correction be? Is scanning worth it if the cleanup is extensive?\n\nIn researching this article, our contractor told me that scanning technology has improved dramatically, even since we began our project a year ago. We hope so. The quality of the conversion we got was so poor that, had we known at the time, we wouldn't have gone the scanning route. Instead we'd have cut back on the data we wanted from the House books and had it double-keyed.",
  "provenance": {
    "extracted_by": "anthropic/claude-sonnet-4-6",
    "extracted_at": "2026-07-05",
    "verified": true
  },
  "similar": [
    {
      "slug": "1995-july-scan-for-the-facts",
      "score": 0.3
    },
    {
      "slug": "1991-november-scanners-may-eliminate-data-entry-problems",
      "score": 0.16
    },
    {
      "slug": "1992-june-correction",
      "score": 0.137
    },
    {
      "slug": "1996-june-errors-can-wreck-stories",
      "score": 0.13
    },
    {
      "slug": "1992-october-more-errors-surface-in-hmda-data",
      "score": 0.121
    }
  ],
  "html_url": "https://dwillis.github.io/uplink/article/1993-january-read-this-before-scanning-text.html",
  "json_url": "https://dwillis.github.io/uplink/article/1993-january-read-this-before-scanning-text.json"
}