{ "id": "2005-septemberoctober-tech-tip-making-xml-useful", "slug": "2005-septemberoctober-tech-tip-making-xml-useful", "url": "https://dwillis.github.io/uplink/article/2005-septemberoctober-tech-tip-making-xml-useful.html", "year": 2005, "month": "September/October", "monthNum": 9, "headline": "Tech tip Making XML useful", "kicker": "", "author_name": "Jeff Porter", "author_title": "", "affiliation": "IRE and NICAR", "summary": "This article explains XML (Extensible Markup Language) and its practical challenges for database users, particularly in government agencies, demonstrating how XML's flexible structure differs from traditional flat files and can be imported into tools like Excel and Access.", "keywords": [ "XML", "database", "government" ], "topics": [ "Technology", "Data & Databases", "Investigative Methods" ], "technologies": [ "XML", "Excel", "Access" ], "full_text": "If you have not had to import XML data into a database manager, you're lucky so far - more and more government agencies use XML to store and transmit information. If the data is relatively small, that's good. But XML files can easily become bloated monsters, frustrating a user of off-the-shelf database software.\n\nWhat is XML? Several definitions are floating around the Web.\n\nExtensible Markup Language is a simple, very flexible text format Originally designed to meet the challenges of large-scale electronic publishing, XML is also playing an increasingly important role in the exchange of a wide variety of data on the Web and elsewhere. (www.advancinginsights.com/mybiz/)\n\nA widely used system for defining data formats. XML provides a very rich system to define complex documents and data structures such as invoices, molecular data, news feeds, glossaries, inventory descriptions, real estate properties, etc. As long as a programmer has the XML definition for a collection of data (often called a \"schema\") then they can create a program to reliably process any data formatted according to those rules. (www.unitedyellowpages.com/internet/terminology.html)\n\nCapable of describing many different kinds of data. Its primary purpose is to facilitate the sharing of structured text and information across the Internet. Languages based on XML (for example, RDF, RSS, MathML, XSIL and SVG) are themselves described in a formal way, allowing programs to modify and validate documents in these languages (http://n.wikipedia.org/wiki/XML)\n\nIf those definitions leave your head spinning (as they did mine), we will focus on the practicality of dissecting XML files - making them useful for data analysis in your software of choice. An XML data file does not have the usual feel of a flat file, such as ASCII and the like. There are no headers, no fixed fields, no columns. Instead, each element is defined line by line, as in this example from the Microsoft support Web site (http://support.microsoft.com).\n\n 1111 Sean Chai NY 1112 Tom Johnston LA 1113 Institute of Art \n\nA bit of translation is in order here.\n\nThe tag (and its counterpart, ) identifies each record that pertains to a given customer. It's repeated for each new record. You can trace the same syntax for CustomerId, CompanyName and City fields. The ROOT tags surround it all. The design can be simple, but in some ways, it's reminiscent of hierarchical data - bringing back the term \"variable length\" - in that each field does not necessarily have to show up in each record.\n\nMicrosoft Excel or Access can import and use many of the XML online files. For practice, download U.S. House roll call voting data that you can get from http://clerk.house.gov/evs/2005/index.asp. For this example, we chose to save this page: http://clerk.house.gov/evs/2005/roll464.xml.\n\nExcel 2003's File > Open menu item has an option to choose XML as file type. After you choose this, it will say \"The file you are opening contains stylesheet(s). What would you like to do:\" Your options are to use an existing stylesheet, ending with the suffix \"xsl\" - in essence, a technical record layout - or opening the file without applying a stylesheet - in other words, relying on the general structure of tags in the file.\n\nWhere does the stylesheet come from? In this instance, it's on the House Web site. Online, the XML file looks nice and pretty. But viewing it with a text editor or using your browser's View Source function, you can easily find a reference for the stylesheet:\n\n \n\n\n\nIf you go to http://clerk.house.gov/evs/vote.xsl, you can view the entire document - a mixture of record layout, code sheet, and tips about the internal data the House uses to create the XML pages.\n\nThat brings us back to the question about whether to use the stylesheet or the file's tags. Which is better? It varies from case to case, so it is a good idea to try both during your first attempt at importing the data.\n\nIn this particular instance, using the stylesheet generates an Excel file that looks suspiciously like the Web page. Go with the other option, pick as an XML list or a read-only workbook and you get what I believe is a better-looking spreadsheet. You actually gain a bit more information, too: party affiliation. Using the stylesheet, parties are identified only by font: Roman type is Republican, italic, Democrat. Independents are underlined. Without stylesheet, the parties are identified with a column containing D for Democrat, R for Republican and I for Independent. And as an added bonus, you get the unique House ID number for each legislator - helpful in joining tables later.\n\nThe loser in this test was Access 2003. You can import the XML using the Import Wizard. Just open or create a database, go to File > Get External Data > Import and choose XML as file type.\n\nBut in this case, the results are disappointing. Like many dataset examples, the XML is structured into several separate \"tables,\" so our results include five tables. The most vital one lists names of legislators and individual votes. Party affiliation and the House IDs are missing.\n\nOther applications are available for working with XML:\n\nXMLSpy from Altova. This program was suggested by several people in and out of journalism. You can download a 30-day evaluation copy from the company Web site at www.altova.com.\n\nNavicat from PremiumSoft. Navicat is a popular graphical interface for MySQL relational database manager that provides a high level of control during importing. A 30-day evaluation copy is available at www.navicat.com.\n\nSAS. This powerhouse stats and data management program (www.sas.com) includes the SAS XML Mapper tool that, in a point-and-click interface, lets the user create a text file of instructions for field names, format and location. You can also edit the file for more control.\n\nPerl. Users of this programming language can download modules to translate XML files.\n\nXML was never designed (in my humble opinion) as a format to store mammoth amounts of data, but increasingly it is being used for large and complex sets of information. Chances are that journalists doing computer-assisted reporting will probably run into a database in this format. Whether it's a huge file, or a small one, there are plenty of tools for CAR practitioners to try.\n\nContact Jeff Porter by e-mail at jeff@ire.org.", "provenance": { "extracted_by": "anthropic/claude-sonnet-4-6", "extracted_at": "2026-07-05", "verified": true }, "similar": [ { "slug": "1997-march-tech-tip", "score": 0.128 }, { "slug": "1995-september-fun-fed-sites", "score": 0.11 }, { "slug": "1998-december-tech-tip", "score": 0.104 }, { "slug": "1994-may-getting-your-program-to-talk-back", "score": 0.103 }, { "slug": "1996-april-parsing-is-the-answer", "score": 0.1 } ], "html_url": "https://dwillis.github.io/uplink/article/2005-septemberoctober-tech-tip-making-xml-useful.html", "json_url": "https://dwillis.github.io/uplink/article/2005-septemberoctober-tech-tip-making-xml-useful.json" }