{
  "id": "2004-march-curl-ing-apart-web-documents",
  "slug": "2004-march-curl-ing-apart-web-documents",
  "url": "https://dwillis.github.io/uplink/article/2004-march-curl-ing-apart-web-documents.html",
  "year": 2004,
  "month": "March",
  "monthNum": 3,
  "headline": "Curl-ing apart Web documents",
  "kicker": "Tech tip",
  "author_name": "Derek Willis",
  "author_title": "Database Editor",
  "affiliation": "Center for Public Integrity",
  "summary": "Curl is a free command-line utility that helps journalists efficiently download large datasets from websites by automating batch downloads of pages with consistent URL patterns, eliminating the need for manual clicking through multiple pages.",
  "keywords": [
    "curl",
    "download",
    "database"
  ],
  "topics": [
    "Investigative Methods",
    "Technology",
    "Data & Databases"
  ],
  "technologies": [
    "curl",
    "ASP",
    "CGI",
    "HTML",
    "Windows",
    "Unix",
    "Linux",
    "OS X"
  ],
  "full_text": "Here's a pet peeve you might recognize: you find a database you need on the Web, but can't download it with a single click. Instead, the records are contained on separate pages, sometimes behind a search form that lets you see only a few entries at a time. Hitting Control-S several hundred times isn't my idea of a great day.\n\nBut there's good news: there are tools that can pry loose all that data without much trouble or effort on your part. My favorite is curl, a command-line utility for Windows, Unix, Linux and Mac OS X. With curl, I can download batches of files with a consistent naming pattern, such as those pesky ASP or CGI files that contain numbers in the URL string.\n\nCurl is a free command-line utility, meaning that you run it from the terminal in Linux, Unix or Mac OS X, or from the DOS prompt in Windows. If your system doesn't have curl installed, you can get it from http://curl.haxx.se. Be sure to bookmark curl's basic documentation (http://curl.sourceforge.net/docs/manpage.html); it comes in handy.\n\nHere's an example of using curl: a few months ago, Jeff South, a Virginia Commonwealth University professor and former database editor at the Austin American-Statesman, posted to the NICAR-L listserv asking if anyone had an easier way to put together a list of California lobbyists.\n\nThe official state Web site (http://calaccess.ss.ca.gov/Lobbying) has a list of current lobbyists but only shows one page for each letter of the alphabet, meaning that you'd have to save each one and then put them together before beginning the task of separating the records from the HTML around them.\n\nCurl was built for just this kind of task, since it offers powerful ways to automate downloads based on a consistent naming pattern. The URL for lobbyists with a last name beginning with",
  "provenance": {
    "extracted_by": "anthropic/claude-sonnet-4-6",
    "extracted_at": "2026-07-05",
    "verified": true
  },
  "similar": [
    {
      "slug": "2001-julyaugust-information-manager",
      "score": 0.153
    },
    {
      "slug": "2004-july-august-pdf-to-text-using-xpdf",
      "score": 0.126
    },
    {
      "slug": "1995-september-nicar-web-site-expands",
      "score": 0.114
    },
    {
      "slug": "1996-september-on-the-internet",
      "score": 0.107
    },
    {
      "slug": "2000-julyaugust-sql-server-pays-off",
      "score": 0.102
    }
  ],
  "html_url": "https://dwillis.github.io/uplink/article/2004-march-curl-ing-apart-web-documents.html",
  "json_url": "https://dwillis.github.io/uplink/article/2004-march-curl-ing-apart-web-documents.json"
}