{
  "id": "1998-march-cleaning-tables",
  "slug": "1998-march-cleaning-tables",
  "url": "https://dwillis.github.io/uplink/article/1998-march-cleaning-tables.html",
  "year": 1998,
  "month": "March",
  "monthNum": 3,
  "headline": "Cleaning tables",
  "kicker": "TECH TIP",
  "author_name": "Andrew Lehren",
  "author_title": "Journalist",
  "affiliation": "Dateline NBC",
  "summary": "A tutorial on writing a FoxPro program to automatically clean up data by standardizing character columns to uppercase and left-aligned formatting, using a scan loop and lookup table approach.",
  "keywords": [
    "FoxPro",
    "data",
    "programming"
  ],
  "topics": [
    "Data & Databases",
    "Investigative Methods",
    "Technology"
  ],
  "technologies": [
    "FoxPro"
  ],
  "full_text": "I don't remember data scrubbing one of the allures of journalism, but it is a perennial topic on the NICAR listserv. All those 'replace all... FoxPro commands can be tiresome, particularly on lengthy tables. But this needs to be done - especially if we want to make sure queries work right.\n\nLast year, at one of NICAR's programming seminars, someone asked whether it's possible to write a FoxPro program that would clean up data automatically with commands that make sure every character column is uppercase and flush left. After a few minutes, I came up with this.\n\nIt's pretty straightforward. For those just learning FoxPro programming, it shows one way to use a scan loop. It relies on just one memory variable. The logic is pretty simple: After some stage setting, the program creates a lookup table so it knows the character fields. Next, the program goes through each character field, one at a time, and runs the 'replace all' commands.\n\nIf you want everything in proper case (with the first letter uppercase and the rest lowercase), you can easily alter this program. This is a stripped-down version to focus on the essence of the program, but you can spruce it up if you're so inclined.\n\nWithout further adieu, let's step through the program:\n\nTo make sure nothing else is open, and that FoxPro starts with a clean slate, we'll begin with these three lines. The first line clears the screen. The second closes any open tables, programs and just about anything else lying about. The third flushes out memory variables that could mess up what follows. Now FoxPro should be pretty clearheaded.\n\nclear close all clear all\n\nWe'll tell FoxPro where to work. When you use this for real, you'll have to adjust by putting in the path for the folder that holds your tables. So change c:\\YourFolder to the path you want.\n\nset directory to \\YourFolder\n\nNext, we'll tell FoxPro what table to clean. You'll need to change TableName. The phrase 'in 1' is just a spot on FoxPro's imaginary desk - like saying I'll put this stack of papers on the left corner so that later, when I open another stack of papers, I'll have some free space on the right side. This makes it easier to go between the two stacks. The 'alias ToClean' means that, from here on out, I'll be calling the table ToClean. This is important because it saves us the trouble of having to edit the rest of the program. You shouldn't need to make any more changes to make this program work for you.\n\nuse TableName in 1 alias ToClean\n\nNext, we create the lookup table. It's just a record layout in a Fox table. That's done with 'copy structure extended.' If this seems bewildering, just run this command on your favorite table and look at the result. It may help you better understand what will follow.\n\ncopy structure extended to Lookup\n\nFoxPro will now open the lookup table in another spot on its imaginary desk.\n\nuse Lookup in 2\n\nHere's the promised memory variable. It will remember the name of every character field.\n\nStore to NameOfField\n\nLet's start going through the lookup table line by line. That's what happens with a scan loop.\n\nselect lookup scan\n\nWe'll check for character fields, and remember their names. Let's say the field name is \"Address,\" and it's a character field. FoxPro now knows that.\n\nif field_type = 'C' Store Lookup.Field_name to NameOfField\n\nTime to flip to our table to be scrubbed.\n\nSelect ToClean\n\nNow we'll make FoxPro run the 'replace all.' The ampersand, by the way, is needed so FoxPro knows we're referring to a memory variable, not a column called literally NameOfField.\n\nreplace all &NameOfField; with alltrim(upper(&NameOfField))\n\nWe're done with that column. Let's see what character columns come next. Will it be a city column? A state column? Whatever it is, we'll be ready, looping through our lookup table one row at a time.\n\nSelect Lookup\n\nThe next two lines close our condition and our scan loop. They're the required periods at the end of our programming thoughts.\n\nEndIf EndScan\n\nBy this point, FoxPro has cleaned all the fields. The next two lines just get our processed table ready to check.\n\nselect ToClean use\n\nWe're done. Open it up and revel in all the typing you've avoided.\n\nAndrew Lehren can be reached at 212-664-7739, or by email at alehren@nbc.com.",
  "provenance": {
    "extracted_by": "anthropic/claude-sonnet-4-6",
    "extracted_at": "2026-07-05",
    "verified": true
  },
  "similar": [
    {
      "slug": "1995-january-editing-data-for-consistency",
      "score": 0.221
    },
    {
      "slug": "1997-november-one-to-many-how-to",
      "score": 0.217
    },
    {
      "slug": "1998-april-foxpro-to-access-and-back-again",
      "score": 0.211
    },
    {
      "slug": "1997-may-cheating-at-foxpro",
      "score": 0.202
    },
    {
      "slug": "1994-june-fixing-data-in-foxpro",
      "score": 0.2
    }
  ],
  "html_url": "https://dwillis.github.io/uplink/article/1998-march-cleaning-tables.html",
  "json_url": "https://dwillis.github.io/uplink/article/1998-march-cleaning-tables.json"
}