{
  "id": "1998-december-tech-tip",
  "slug": "1998-december-tech-tip",
  "url": "https://dwillis.github.io/uplink/article/1998-december-tech-tip.html",
  "year": 1998,
  "month": "December",
  "monthNum": 12,
  "headline": "TECH TIP",
  "kicker": "",
  "author_name": "Steve Doig",
  "author_title": "",
  "affiliation": "Arizona State University",
  "summary": "Computer-assisted reporting professionals tackle messy, unstructured datasets using various tools and techniques, with the NICAR-L community providing valuable collaborative problem-solving resources and demonstrating multiple effective approaches to data challenges.",
  "keywords": [
    "data",
    "journalism",
    "programming"
  ],
  "topics": [
    "Investigative Methods",
    "Technology",
    "Data & Databases"
  ],
  "technologies": [
    "SAS",
    "Access",
    "FoxPro",
    "Visual Basic",
    "Excel",
    "Monarch",
    "Perl",
    "CD-ROM",
    "magnetic tape"
  ],
  "full_text": "A big part of the fun for most CAR folks is the challenge of solving puzzles. projects tend to be one-of-a-kind things, so we're constantly facing new data formats with strange layouts (at least to us), undocumented codes and other idiosyncrasies. Even though no one else in the newsroom understands how difficult the struggle was, we get a rush of satisfaction from finally beating an unruly dataset into submission.\n\nWhen tackling such data puzzles, a great resource is the NICAR-L list. Even if you've never faced variable-length records or packed hexadecimals or YYYYDDMM dates or bitmaps or other weirdness, chances are good that others have. And perhaps the most instructive thing about the flurry of postings often generated by an interesting question is the wide range of clever solutions offered. Over and over, we discover that in CAR there are a lot of ways to get from here to there.\n\nA case in point was the \"ugly data\" contest I tossed onto NICAR-L this summer. It not only provided a stimulating puzzle for some of our best propeller-heads, but the results also proved the versatility of the CAR toolbox.\n\nI found this puzzle in a big tape dataset sent to me by a reporter who needed it in a more useful format and on CD-ROM. The data, which came as a courts database involving civil suits, offered several kinds of ugliness:\n\nThe structure was hierarchical: For each case there was one case record and then as many individual party records as there were parties to the case. But the case record didn't say how many party records would follow. Nor was there a common field, like case number, to link case and parties.\n\nEach \"line\" of the data was composed of exactly 100 of the above case and party records, with no delimiters between records. Thus, a \"line\" might be anywhere from 4,500 to about 6,000 characters long. If the last case still had more parties when the 100-record limit was reached, the rest would spill across to a new \"line.\"\n\nThe layout we were given was wrong. It didn't mention that each case record began with \"0026\" and each party record with \"0055.\" Later, I figured out that this undocumented feature was supposed to be the length of each record type, though why the file creators felt the need to repeat this for every record is beyond me.\n\nNor were the supposed record lengths of 26 or 55 characters reliable. Some, but not all, records ended with one, two or three carets, for no apparent reason.\n\nIt took me an afternoon of noodling with SAS, my database program of choice, to figure out a way to read the data and transform it into something useful. I've been on a long campaign to convince CAR power users that SAS is better than more commonly used programs such as Access or FoxPro, particularly for ugly data like this. So I created the \"Ugly Data Contest,\" urging anyone interested to tackle it with whatever tools they wanted. I was sure their struggles, and the relative simplicity of my SAS program, would win some converts.\n\nThe roads to solution\n\nI won't go into all the details of the results here (you can check the NICAR-L archives of August 4-10, 1998, by going online at http://www.nicar.org/nicarl.html). But I was surprised and delighted - despite my pro-SAS agenda - at the wide variety and inventiveness of the solutions that came in.\n\nAlong with my SAS program, there were FoxPro programs from Greg Reeves of the Kansas City and Griff Palmer of the Daily Oklahoman; a Visual Basic solution from David Heath of the St. Louis Post-Dispatch; a stand-alone Visual Basic GUI from Dan Keating of The Miami Herald; a Word-and-Excel method-in-concept (that I called the \"ugly solution\" winner!) from Russ Clemings of the Fresno Bee; a Visual FoxPro/SQL approach from Jeff Porter of the Arkansas Demo-Gat-Gazette, and a text editor/Monarch solution from Dick Eijk of NRC Handelsblad in Rotterdam, The Netherlands.\n\nBut my favorite solution used an entirely unexpected tool: PERL, the \"print extraction and report language\" loved by Unix wizards and cgi-scripters. Matt Ericson of the Philadelphia Inquirer's web operation did it with these five lines:\n\nPut a new line after any party record\nperl -i -pe 's/ (\\d|d\\d\\d\\d\\d) (\\d\\d\\d\\d) ([CP]) / \\1\\n\\3/g' ug.txt\n\nPut a new line after any case record\nperl -i -pe 's/(\\d\\d\\d\\d\\d\\d\\w\\w\\w\\w\\w\\w) (\\d\\d\\d\\d) ([CP])/\\1\\n\\3/g' ug.txt\n\nFix any line that still starts with a 4 digit number, which seems to appear after the ^s\nperl -i -pe 's/^\\d\\d\\d\\d//;' ug.txt\n\nKill any ^s\nperl -i -pe 's/\\^//g;' ug.txt\n\nMerge the cases and parties\nperl -i -pe 'if (/^C/) { $c=$_=\"=\"; chop ($c) } else $_=\"$c ug.txt\n\nI still say SAS is best for handling the widest range of ugly data and other CAR problems. (I am, however, starting to learn some PERL!) But the contest showed it's possible to make almost any CAR tool do almost any CAR task. As the old saying goes: \"When all you have is a hammer, every problem looks like a nail.\"\n\nSteve Doig can be reached by e-mail at steve.doig@asu.edu",
  "provenance": {
    "extracted_by": "anthropic/claude-sonnet-4-6",
    "extracted_at": "2026-07-05",
    "verified": true
  },
  "similar": [
    {
      "slug": "1992-september-problem-solving-at-us-news-and-world-report",
      "score": 0.185
    },
    {
      "slug": "2003-novemberdecember-readmetxt-car-tool-tip",
      "score": 0.153
    },
    {
      "slug": "1997-march-tech-tip",
      "score": 0.138
    },
    {
      "slug": "2006-november-december-hard-lessons-in-first-big-data-wave",
      "score": 0.133
    },
    {
      "slug": "2000-januaryfebruary-the-future-of-car",
      "score": 0.131
    }
  ],
  "html_url": "https://dwillis.github.io/uplink/article/1998-december-tech-tip.html",
  "json_url": "https://dwillis.github.io/uplink/article/1998-december-tech-tip.json"
}