{
  "id": "2004-september-october-scaping-for-data-the-perl-way",
  "slug": "2004-september-october-scaping-for-data-the-perl-way",
  "url": "https://dwillis.github.io/uplink/article/2004-september-october-scaping-for-data-the-perl-way.html",
  "year": 2004,
  "month": "September • October",
  "monthNum": 9,
  "headline": "Scaping for data the Perl way",
  "kicker": "Tech tip...",
  "author_name": "John Perry",
  "author_title": "",
  "affiliation": "The Oklahoman",
  "summary": "John Perry explains how to use Perl scripts and specific modules like WWW-Mechanize to automate the collection of data from court docket websites.",
  "keywords": [
    "Perl",
    "web scraping",
    "automation"
  ],
  "topics": [
    "Technology",
    "Investigative Methods"
  ],
  "technologies": [
    "Perl",
    "WWW-Mechanize",
    "HTML-TokeParser",
    "PHP",
    "Python",
    "Curl"
  ],
  "full_text": "In the March/April edition of Uplink, Derek Willis wrote about using the free utility Curl to automate the task of downloading pages from Web sites. Sometimes you need a smarter tool that cannot only find Web pages, but also looks at the content and decides what to do with it. That's where Perl – and other scripting languages such as PHP and Python – come to the rescue. Two modules, WWW-Mechanize and HTML-TokeParser, are godsends for Web scrapers. WWW-Mechanize acts like a virtual Web browser and hides the technical intricacies of HTTP requests. HTML-TokeParser is one of several tools Perl provides for pulling useful information from Web pages. Web scraping came in handy during a recent investigation of the Oklahoma County District Court and the court clerk's office. We wanted to look at all Oklahoma County cases in which the bond had been forfeited. The Supreme Court's Web search result pages are simple and seem designed for easy parsing. With a Perl script using the Mechanize module, I grabbed the docket info page for each of the cases and saved them in a directory. I then wrote a script that used the TokeParser module to search for key words in the docket text. The first step in Web scraping is to find the information about the search page that you'll need. To find and save all the cases in our list of bond forfeitures, we had to accomplish these tasks: get the search form page; set the 'db' and 'number' fields; click the search button; save the page; and return to the form. A one-second sleep is necessary because servers protect themselves from denial of service attacks. TokeParser turns a Web page into a stream of tokens. To get to the docket text, we start at a comment, then step down rows and cells. This is a relatively simple example of the capabilities of Perl and its collection of Web scraping tools.",
  "provenance": {
    "extracted_by": "pre-v2",
    "extracted_at": "",
    "verified": true
  },
  "similar": [
    {
      "slug": "2000-june-the-perl-y-gates",
      "score": 0.433
    },
    {
      "slug": "2004-september-october-scraping-away-for-useful-data",
      "score": 0.414
    },
    {
      "slug": "2000-julyaugust-more-on-perl",
      "score": 0.232
    },
    {
      "slug": "2004-november-scrape-away",
      "score": 0.208
    },
    {
      "slug": "2000-januaryfebruary-parsing-nightmare",
      "score": 0.149
    }
  ],
  "html_url": "https://dwillis.github.io/uplink/article/2004-september-october-scaping-for-data-the-perl-way.html",
  "json_url": "https://dwillis.github.io/uplink/article/2004-september-october-scaping-for-data-the-perl-way.json"
}