{
  "id": "2000-october-programs-for-parsing",
  "slug": "2000-october-programs-for-parsing",
  "url": "https://dwillis.github.io/uplink/article/2000-october-programs-for-parsing.html",
  "year": 2000,
  "month": "October",
  "monthNum": 10,
  "headline": "Programs for parsing",
  "kicker": "TECH TIP",
  "author_name": "David Heath",
  "author_title": "",
  "affiliation": "Seattle Times",
  "summary": "The article explains how to handle difficult-to-import data files using programming, specifically Visual Basic, when traditional tools like Excel's import wizard fall short. It outlines five key programming concepts needed to parse loosely structured data and recommends learning resources for getting started.",
  "keywords": [
    "programming",
    "data",
    "parsing"
  ],
  "topics": [
    "Investigative Methods",
    "Technology",
    "Data & Databases"
  ],
  "technologies": [
    "Visual Basic",
    "FoxPro",
    "Perl",
    "Excel",
    "Access",
    "Word",
    "Monarch",
    "QBasic",
    "HTML"
  ],
  "full_text": "If you work with data much, at some point you're going to deal with files that defy easy import.\n\nKnowing a few tricks can sometimes save you lots of work. For example, if a text file is missing carriage returns at the end of each line — a common dilemma — you can open it in Word and save it into a new text file. Voilà. You now have carriage returns.\n\nYou can also do a lot with search and replace tools, Excel's import wizard and Monarch, a terrific software package that converts reports back into databases. But there are times when even Monarch falls short. That's especially true if columns don't line up or the structure of each record is not consistent.\n\nWhen the going gets rough, the tough write a program.\n\nWriting your own program is the only solution I know if you're trying to parse data that's loosely structured. Say that you want to take a collection of Web pages and put all the links into a database. The links aren't hard to find; they always begin with \"<a href=\" But otherwise, the pages may lack any consistent format.\n\nThere is a way to do this, but you have to learn a smidgen of programming. If you already write simple programs to clean data, you probably already have most of the skills needed. If you are a novice or have never tried programming, the good news is that you don't have to learn that much. You'll be working with a limited set of functions and statements.\n\nI'm going to show you how to deal with problems using Visual Basic. You could do the same type of thing with FoxPro or, for those fearless of its steep learning curve, Perl. The latter is especially powerful at pattern matching, making it an ideal tool for finding HTML tags.\n\nVB's chief advantages are that it's already built in to Access, Excel and Word — so you don't have to buy any new software — and it was especially created to be an easy-to-learn programming language. (Although you've already got a version of VB, if you buy the full version you'll get a lot of debugging tools that can be a life saver.)\n\nTo manipulate text in VB, you need to learn five things:\n\n1. How to declare variables, which simply means defining them as either characters or numbers.\n\n2. How to open and close files.\n\n3. How to read from files and write to files.\n\n4. How to use text functions, such as MID, INSTR, LEFT, TRIM and LEN\n\n5. How to create loops.\n\nWhen you're first learning Visual Basic, it's easy to get distracted making pretty pictures: forms, buttons, labels, etc. But your program doesn't have to look good; it just needs to get the job done. In fact, all the tools you need can be found in the old QBasic, that free program that comes with DOS. A great book for learning Basic programming is \"QBasic by Example,\" written by Greg M. Perry.\n\nThere's no way to teach you everything you'll need to start programming in a brief article, so I highly recommend getting a book. However, I can walk you through a fairly simple program that parses data, and that should give you a pretty good idea of what to expect.\n\nLet's take the U.S. Census home page, at www.census.gov, and parse all the links into a comma-delimited file that we can easily import into a database. Save the page to your hard drive and note the location, because you'll need it for later. The file is tricky because the page is very loosely structured and the HTML tags are not always consistent. Sometimes they are followed by quotation marks and sometimes they aren't. So we have to include some conditional statements in our programs.\n\nOf course, it's not efficient to write a program to parse a single page, but this program could parse scores of Web pages all at once. You'll find plenty of situations where such a program can do wonders.\n\nTo run this, go to Tools | Macro | Visual Basic Editor in Excel and type in the following code. Then simply click on the Run button.\n\nPrivate Sub Command1_Click()\n\nStart by declaring the data type of all the variables you'll use.\n\nI've just made up the variable names on the left, starting some with \"str\" to signify string, int to signify integer or lng to signify Long Integer.\n\nDim strLine As String\nDim intRead, intWrite As Integer\nDim lngFindAHREF, lngEndHere, lngFindAlt As Long\nDim lngStartHere, lngFindEndTag, lngFindName As Long\nDim lngFindGreaterThan, lngFindLessThan As Long\nDim strHref, strAlt, strName As String\nDim strNextLine As String\n\nFreeFile() is a useful function that assigns an unused number to a file. Basic requires files to be numbered for opening, closing, reading and writing.\nintRead = FreeFile()\n\nIn this case, intRead is now equal to 1. We can now open our Web page to read from as file No. 1.\nOpen \"c:\\census\\www_census_gov.html\" For Input As #intRead\n\nNext we want to create a second file, No. 2, that we are going to write the data to.\nintWrite = FreeFile()\nOpen \"c:\\census\\filetoimport.txt\" For Output As #intWrite\n\nSo all we've done is opened the Web page, which was previously saved to your computer. Your path may differ from that above. Then we created a file called \"filetoimport.txt\" that we will write to.\n\nWe're now ready to look through the Web page for patterns. The statement for that is\nDo while NOT EOF(intRead)\nMeaning read until you get to the End Of File #1, followed by\nLine Input #intRead, strLine,\nmeaning\nGo to the next line and put it into my variable \"strLine.\"\nDo While Not EOF(intRead)\nLine Input #intRead, strLine\n\nTime to start looking for the pattern \"a href\". The function InStr gives the number of the characters from the left that a pattern appears.\n\nvbTextCompare _\nignores upper and lower cases.\nlngFindAHREF = InStr(1, strLine, \"A HREF\", vbTextCompare)\n\nIf we find \"a href\" in a line, we want to grab that line eight characters to the right of the start of the tag.\nIf lngFindAHREF > 0 Then strLine = Mid(strLine, lngFindAHREF + 8)\n\nSome of the lines wrap, so we have to make them one long line using a loop.\nDo While Not InStr(1, strLine, \"</A\", vbTextCompare) > 0\nLine Input #intRead, strNextLine\nstrLine = LTrim(strLine) + \" \" + LTrim(strNextLine)\nLoop\n\nWe're looking for the end of the first tag and will put the text in between into a variable called strHref.\nlngEndHere = InStr(1, strLine, \">\", vbTextCompare)\nstrHref = Mid(strLine, 1, lngEndHere - 1)\n\nNow we're looking for the ALT tag, which is what shows up if an image can't be displayed on the Web page. If we find it, we'll put the text in that tag into the variable \"strAlt.\" To help find the end of the tag, we're searching for quotation marks. I've used the Chr() function because we can't put quotations within quotations.\nlngFindAlt = InStr(1, strLine, \"ALT\", vbTextCompare)\nIf lngFindAlt > 0 Then\nlngStartHere = InStr(lngFindAlt, strLine, Chr(34)) + 1\nlngEndHere = InStr(lngStartHere, strLine, Chr(34))\nstrAlt = Mid(strLine, lngStartHere, lngEndHere - lngStartHere)\nEnd If\n\nFinally, we're looking for the last tag, which sometimes will include text that you see on the Web page as a hyperlink.\nlngFindEndTag = InStr(1, strLine, \"</a\", vbTextCompare)\nstrLine = Mid(strLine, 1, lngFindEndTag 1)\n\nlngFindGreaterThan = InStr(1, strLine, \">\")\nlngFindLessThan = InStr(1, strLine, \"<\")\nIf lngFindLessThan > 0 Then\nstrLine = Mid(strLine, lngFindGreaterThan + 1)\nEnd If\nlngFindName = InStr(strLine, Chr(62))\nstrName = Mid(strLine, lngFindName + 1)\n\nWe have a little bit of clean up work to do. There are extraneous quotation marks to remove and pieces of unwanted text. Also, some of the links begin with \"http://\" and some don't.\nIf Left(strHref, 1) = Chr(34) Then strHref = Mid(strHref, 2)\nIf InStr(1, strHref, \" \") > 0 Then strHref = Mid(strHref, 1, InStr(1, strHref, \" \") - 2)\nSelect Case Left(strHref, 1)\nCase \"h\"\nstrHref = strHref\nCase \"/\"\nstrHref = \"http://www.census.gov\" + strHref\nCase Else\nstrHref = \"http://www.census.gov/\" + strHref\nEnd Select\nIf Right(strHref, 1) = Chr(34) Then strHref =\nMid(strHref, 1, Len(strHref) - 1)\n\nWe're combing the ALT tags with the hyperlinks because they do essentially the same thing.\nIf strAlt = \"\" Then strAlt = strName\n\nNow we'll write the variables to a comma-delimited file.\nWrite #intWrite, strHref, strAlt\n\nIt's time to clean out the variables and run through the loop again.\nstrHref = \"\"\nstrAlt = \"\"\nstrName = \"\"\nEnd If\nLoop\n\nYou always have to close files that you've opened. The command for that is simple.\nClose\nEnd\nEnd Sub\n\nDavid Heath can be reached by e-mail at dheath@seattletimes.com.",
  "provenance": {
    "extracted_by": "anthropic/claude-sonnet-4-6",
    "extracted_at": "2026-07-05",
    "verified": true
  },
  "similar": [
    {
      "slug": "1996-april-parsing-is-the-answer",
      "score": 0.303
    },
    {
      "slug": "1993-augustseptember-programming-as-easy-as-sql",
      "score": 0.244
    },
    {
      "slug": "2000-october-vbasic-resources",
      "score": 0.239
    },
    {
      "slug": "2000-januaryfebruary-parsing-nightmare",
      "score": 0.222
    },
    {
      "slug": "2000-june-the-perl-y-gates",
      "score": 0.216
    }
  ],
  "html_url": "https://dwillis.github.io/uplink/article/2000-october-programs-for-parsing.html",
  "json_url": "https://dwillis.github.io/uplink/article/2000-october-programs-for-parsing.json"
}