{ "id": "2000-june-the-perl-y-gates", "slug": "2000-june-the-perl-y-gates", "url": "https://dwillis.github.io/uplink/article/2000-june-the-perl-y-gates.html", "year": 2000, "month": "June", "monthNum": 6, "headline": "The Perl-y gates", "kicker": "PROGRAMMING", "author_name": "Matthew Ericson", "author_title": "Reporter", "affiliation": "The Philadelphia Inquirer", "summary": "A programming competition demonstrates Perl's superior speed and efficiency for data parsing compared to VBScript, highlighting why Perl has become an essential tool for computer-assisted reporting and web development.", "keywords": [ "Perl", "programming", "parsing" ], "topics": [ "Investigative Methods", "Technology", "Data & Databases" ], "technologies": [ "Perl", "Visual Basic", "SQL Server", "ODBC", "ADO", "Unix" ], "full_text": "Ok, it wasn't as thrilling as the chariot race in Ben Hur, but we found it exciting anyway. Tom Torok, then The Inquirer's Visual Basic guru, and I, a Perl fanatic, squared off to see whose programming language could most quickly parse a report into a tab-delimited format. The results, as reported by Torok to the NICAR mailing list: \"Matt Ericson and I just had a Perl vs. VBScript race on parsing a 130 meg report-form file into a tab-delimited file suitable for import into SQL Server.\n\nThe outcome: Matt (and Perl) beat the pants off Torok (and VBScript). His Perl script parsed nearly 750,000 records in less than half the time (with less than half the code.)\n\nProgrammers discovered Perl was a great language to use when creating dynamic web pages - it's also a great tool for cleaning up data in computer-assisted reporting.\n\nI'm not only sold, but I also have to dress like a chicken for a week.\"\n\nPerl - \"Practical Extraction and Report Language\" or \"Pathologically Eclectic Rubbish Lister\" - is a programming language foreign to most users of Microsoft Windows. It started as a utility to read text files and produce reports on Unix-based computer systems. With the growth of the World Wide Web, programmers discovered it was a great language to use when creating dynamic Web pages.\n\nHere's a few reasons it works so well:\n\nIt's also a great tool for cleaning up data in the computer-assisted reporting world.\n\nQuick, free and competent\n\nPerl has a wide range of built-in functions for dealing with data in many formats-fixed-width text files, delimited text files, queries from databases - and then dealing efficiently with the data. You don't have to worry about whether you've defined your variables as the right types - all variables are the same type in Perl - so you can use the same variable once as a number and later as a string. Perl doesn't care - it converts it transparently. A string that grows longer than 255 characters? Perl will expand it on the fly. The same goes for arrays - they can grow as big as needed. And while it's not as fast a language as C, it more than holds its own with Visual Basic. And, unlike C, you can write useful, short programs quickly and easily. Plus, it's free. How can you beat that? (Download it from www.perl.com.)\n\nData in many ways\n\nPerl has myriad ways to read data in text files. By default, it reads line-by-line. But if a record spans multiple lines, you can change the end-of-record delimiter. Or, read the whole file in one fell swoop.\n\nIs the data tab-delimited? Then use Perl's split function to break a record into its respective fields.\n\nDelimited by something other than tabs? The split will handle that, too. Just tell it what character (or characters) is the delimiter.\n\nIs the data fixed-width? Use the unpack function to pull it apart.\n\nPerl has drivers to access most types of databases. There are drivers for Microsoft SQL Server, Oracle, Sybase and others.\n\nPlus, if you're using Perl on Microsoft Windows, you can use ODBC and ADO just like you can in Visual Basic (with a few slight syntax changes). And, with SQL Server, Perl is an option for writing Data Transformation Services scripts.\n\nPowerful search and replace\n\nPerl's strongest feature is its use of \"regular expressions,\" a fancy Unix term for an extremely powerful search-and-replace mechanism.\n\nUnlike most programs, where search-and-replace means searching for a word and replacing with another word, regular expressions let you search for patterns of characters.\n\nIt can, for example, find a pattern of 2 digits followed by a letter and then use pieces of that search in the replacement - taking that letter and putting it before the 2 digits, for example.\n\nPowerful associative arrays\n\nAnother unusual Perl feature is its associative arrays - arrays that are indexed off of a string, rather than a number.\n\nMost programming languages let you store data in arrays: you put one piece of data in the first element of the array, another piece of data in the second element, and so on. Associative arrays, also called \"hashes,\" are arrays that are indexed not on integers, but on arbitrary string values.\n\nAssociative arrays: For example, the array here is what you might store in an array named \"color:\" $color{\"apple\"} = \"Red\"; $color{\"banana\"} = \"Yellow\"; $color{\"grape\"} = \"Purple\"; If you are accessing a database of fruit, you can find the color of the fruit by looking in the associative array. If the current record in the database is for an apple, and has a field named name_of_fruit, then $color{$name_of_fruit} will return \"Red.\"\n\nAssociative arrays let you build lookup tables on the fly, and you can also use associative arrays to count things. If you are parsing a text file containing street addresses, you can keep track of how often a street appears in the file by adding 1 to the array for each street name:\n\n($count{$street} = $count{$street} + 1) When you're done with the file, to find out how often Broad Street appeared, print out $count{'BROAD STREET'}. Sure, it's not going to replace Group By queries in SQL, but for quick-and-dirty counting, it can't be beat.\n\nIt flies on the Web\n\nMany of the same functions that make Perl superb for data manipulation also make it a great language for generating dynamic Web pages. On Microsoft Windows computers, Perl can be embedded in Active Server Pages just like VBScript. Plus, you get access to all of the ASP objects - the Request objects to let you process data from an HTML form, the Server objects to open connections to ODBC and ADO databases, and so forth.\n\nThe bad news\n\nWhat's the catch? The most intimidating thing to Perl newcomers is its cryptic syntax. If you've grown up with C and Unix utilities, Perl will seem right at home. However, if you're coming from a DOS and Visual Basic background, the abundance of strange characters - all those tildes, curly braces, carets, slashes, backslashes and so forth be downright befuddling. You end up with commands such as: foreach $a @data ) { $a s/^ +//; This makes perfect sense to Perl programmers, but resembles nothing you ever saw in Visual Basic. (FYI: that command removes any extra spaces from the start of each element in an array.) It's worth it, though. Take the time to learn Perl and soon you'll find yourself cringing when someone suggests you write a script or Web page in Visual Basic. (Especially when you find that one-line Perl script can sometimes do more than 20 lines of VB.)\n\nMore resources:\n\nFor beginners using Microsoft Windows systems, Learning Perl on Win32 Systems, by Randal L. Schwartz, offers a good place to start. It's published by O'Reilly and Associates.\nThe bible of the language, Programming Perl, was written by Larry Wall, the language's creator, and is probably the most complete guide to the language. Also published by O'Reilly, it's commonly known as \"the camel book\" because of the picture of the camel on the cover. Compared to Learning Perl, it's less of a tutorial and more of a reference manual. While it assumes some Unix familiarity, it's still excellent for users who haven't used Unix.\n\nMatthew Ericson can be reached by e-mail at mericson@phillynews.com", "provenance": { "extracted_by": "anthropic/claude-sonnet-4-6", "extracted_at": "2026-07-05", "verified": true }, "similar": [ { "slug": "2004-september-october-scaping-for-data-the-perl-way", "score": 0.433 }, { "slug": "2000-januaryfebruary-parsing-nightmare", "score": 0.32 }, { "slug": "2000-julyaugust-more-on-perl", "score": 0.302 }, { "slug": "2004-september-october-scraping-away-for-useful-data", "score": 0.221 }, { "slug": "2000-october-programs-for-parsing", "score": 0.216 } ], "html_url": "https://dwillis.github.io/uplink/article/2000-june-the-perl-y-gates.html", "json_url": "https://dwillis.github.io/uplink/article/2000-june-the-perl-y-gates.json" }