{
  "id": "1992-december-getting-rid-of-those-pesky-duplicate-records",
  "slug": "1992-december-getting-rid-of-those-pesky-duplicate-records",
  "url": "https://dwillis.github.io/uplink/article/1992-december-getting-rid-of-those-pesky-duplicate-records.html",
  "year": 1992,
  "month": "December",
  "monthNum": 12,
  "headline": "Getting rid of those pesky duplicate records",
  "kicker": "Tech tips",
  "author_name": "Elliot Jaspin",
  "author_title": "",
  "affiliation": "Cox Newspapers",
  "summary": "This article explains how to eliminate duplicate records in a database using programming languages, comparing the process to manually sorting through papers and removing duplicates by matching key fields and deleting matching second records.",
  "keywords": [
    "database",
    "duplicates",
    "programming"
  ],
  "topics": [
    "Data & Databases",
    "Investigative Methods",
    "Technology"
  ],
  "technologies": [
    "XDB",
    "Paradox",
    "dBase"
  ],
  "full_text": "One of life's more vexing problems - along with faulty plumbing and telephone solicitations - is getting rid of duplicate records. You can, of course, use the DISTINCT command, but it has its limitations. If you want to eliminate the second record with a duplicate field in a list, DISTINCT has no way of distinguishing between which record came first and which second.\n\nThe only alternative, short of manually deleting each record, is to use the programming language that is part of any database software. While the following example is done using the procedural language in XDB, the same approach can be modified to work with PAL in Paradox or any of the xbase products.\n\nBefore writing any code to accomplish this task, imagine how you might do this if you were working with a stack of papers instead of electronic records. You would first sort the records to make sure that the first record you wanted of any duplicates would always appear first. Next you would take the first record off the stack, and compare a key with the second record. If the two fields matched you would throw away the second record. But if they didn't match, you would save the second record and use it to match against the record. You would then repeat the process until you had gone through all the records.\n\nWhile the instructions to the computer will appear different from the way it is described above, the method is the same.\n\nLet us assume our list of records looks like the following table, called Doubles:\n\nKey\tCompany\tLast\tFirst\n57\tAjax\tSmith\tJohn\n44\tWidget\tClinton\tBill\n44\tWidget\tRoebuck\tBubba\n32\tWidget\tWheedle\tSkeeter\n88\tRebar\tLybdenum\tMolly\n88\tRebar\tStang\tArnold\n\nThe first step is to select a stack of records and sort them according to our criteria. In XDB this means running a select statement and assigning the result to a label.\n\nQ1: SELECT FROM DOUBLES ORDER BY KEY, COMPANY, LAST\n\nJust as with our hypothetical stack, you then select the first record and load the comparison value into a variable.\n\nFIRST Q1 < This selects the first record, and COMPVAL = Q1.KEY < the value in the key field is loaded into compval.\n\nOur next step is to select the second record from the list before we begin repeating the process.\n\nNEXT Q1\n\nTo loop through the entire list we can use a WHILE statement. In this case the WHILE statement is constantly checking to see if the next record grabbed from the list is the last record in the list. In XDB, determining the last record is done by checking the \"stat\" value. If STATUS is \"1\" then there are still more records. But if the status value changes to \"0\" we know we have reached the end of the list. What we are saying in the next statement is to keep taking records from the list until there are no more records - until the value of status is \"0.\"\n\nWHILE (STATUS[Q1])>0\n\nWe're now ready to make the actual comparison, using an \"if-then-else\" statement.\n\nIf the two values match then delete the second record, grab another record, make it the comparison value and repeat the process. ELSE makes the second record the comparison value.\n\nIF COMPVAL = Q1.KEY If they match, DELETE Q1 < delete the second record, NEXT Q1 < get another record COMPVAL = Q1 KEY < make it the comparison value. ELSE COMPVAL = Q1 KEY < Otherwise make the ENDIF second record the comparison value.\n\nWhether we decide to delete a record, the last thing we need to do is get another record.\n\nNEXT Q1\n\nTo repeat the whole process we end this short routine with an ENDWHILE. That forces our program to loop back to the WHILE statement and repeat the process until, of course, we run out of records.\n\nNote that this method only deletes the first duplicate record. If more than one duplicate exists, another approach must be used.\n\nNow if we could just write a program to prevent salesmen from hawking life insurance by phone.",
  "provenance": {
    "extracted_by": "anthropic/claude-sonnet-4-6",
    "extracted_at": "2026-07-05",
    "verified": true
  },
  "similar": [
    {
      "slug": "2002-november-december-checking-for-duplicates-in-excel",
      "score": 0.3
    },
    {
      "slug": "1997-november-one-to-many-how-to",
      "score": 0.291
    },
    {
      "slug": "1993-augustseptember-programming-as-easy-as-sql",
      "score": 0.229
    },
    {
      "slug": "2000-october-programs-for-parsing",
      "score": 0.177
    },
    {
      "slug": "1994-march-how-to-keep-data-straight-by-adding-an-id",
      "score": 0.172
    }
  ],
  "html_url": "https://dwillis.github.io/uplink/article/1992-december-getting-rid-of-those-pesky-duplicate-records.html",
  "json_url": "https://dwillis.github.io/uplink/article/1992-december-getting-rid-of-those-pesky-duplicate-records.json"
}