{
  "id": "1999-julyaugust-stats-and-such-raging-to-regress",
  "slug": "1999-julyaugust-stats-and-such-raging-to-regress",
  "url": "https://dwillis.github.io/uplink/article/1999-julyaugust-stats-and-such-raging-to-regress.html",
  "year": 1999,
  "month": "July/August",
  "monthNum": 7,
  "headline": "STATS AND SUCH: Raging to regress",
  "kicker": "STATS AND SUCH",
  "author_name": "Neill A. Borowski",
  "author_title": "",
  "affiliation": "The Philadelphia Inquirer",
  "summary": "A journalism educator explains the mathematical foundations of linear regression analysis, encouraging reporters to understand the statistical processes behind regression software rather than treating it as a black box. The article walks through the method of least squares using a practical example comparing school free lunch percentages to test scores.",
  "keywords": [
    "regression",
    "statistics",
    "journalism"
  ],
  "topics": [
    "Data & Databases",
    "Investigative Methods",
    "Journalism & Newsroom"
  ],
  "technologies": [
    "SPSS",
    "SAS",
    "Excel"
  ],
  "full_text": "How many times have you interviewed an \"expert\" who quickly described a process without going into detail? If you're any kind of interviewer, you made this expert stop and painstakingly explain how the process worked. The process might have been a controversial bond issue, a new communications technology or the treatment for a disease.\n\nNow let's consider the rage to regress. Only at a NICAR session will you hear journalists in the bar late at night discussing how they \"ran this independent variable with that dependent variable and came up with an r-square of\n\nYet, when we use this popular and insightful tool, we never press for an explanation of how the process works. We plug our variables into the black box (SPSS, SAS, Excel, etc.) and out pops the regression results. Many don't seem the least bit interested in how the black box did what it did.\n\nCall me a math nerd, but I like to understand the statistical underpinnings. Obviously you can't hand-crank a regression analysis of thousands of records. But it still makes sense to understand the process that's going on inside the black box.\n\nOur goal in simple linear regression analysis is to come up with a model: + random error. The \"y\" is your dependent variable (such as test scores). The \"X\" is your independent variable (such as poverty rates). The \"a\" is the slope of the regression line and the \"b\" is the constant (the intercept of the vertical axis). This is the equation of the regression line.\n\nThe regression line is fitted from, in essence, many averages. The model tries to plot the line that best fits the data that you've provided. The first step is a scatterplot, simply taking each data point and drawing a point where its X value and y value intersect. The regression line drawn through the points is the best fit, based on the average lengths of the data points from the regression line.\n\nThese two charts represent the percentage of students eligible for free lunch in a school (x) and that school's average test score (y). The other columns simply are X-squared, y-squared and X times y.\n\nFREE LUNCH % (x) | x2 | TEST SCORE (y) | y | xy\n0.55 | 0.3025 | 900 | 810000 | 495\n0.67 | 0.4489 | 900 | 810000 | 603\n0.03 | 0.0009 | 1365 | 1863225 | 40.95\n0.48 | 0.2304 | 1152 | 1327104 | 552.96\n0.12 | 0.0144 | 1218 | 1483524 | 146.16\n0.89 | 0.7921 | 856 | 732736 | 761.84\n0.76 | 0.5776 | 1010 | 1020100 | 767.6\n0.08 | 0.0064 | 1300 | 1690000 | 104\n0.26 | 0.0676 | 1193 | 1423249 | 310.18\n0.35 | 0.1225 | 1297 | 1682209 | 453.95\nSUMS\n4.19 | 2.5633 | 11191 | 12842147 | 4235.64\n\nNow the regression math - or \"method of least squares\" - begins.\n\nSxx = (The sum of the X squares) - (The sum of the x's, squared/n)\n\nOr, in math notation:\n\nSxx = - Ex-ex\n\nSubstituting from the table above:\n\nSxx = 0.80769\n\nNow the same for the y's (test scores):\n\nSyy = (The sum of the y squares) - (The sum of the y's, squared/n)\n\nOr, in math notation:\n\n=\n\nSubstituting from the table above:\n\nSyy = 12,842,147-11,191°-318,2\n\nFinally, we need a third \"average\" - the average of the x's times the y's.\n\nSxy = (The sum of the xs times the ys) - (The sum of the x's times the sums of the y's)/n\n\nOr, in math notation:\n\n=\n\nSubstituting from the table above:\n\n=\n\nWith these \"averages\" behind us, we now can find the components of the regression line.\n\n=-561.34\n\nSince a is the slope of the regression line, and because a is negative, the line is negatively sloped - meaning that it goes from the upper left to the lower right.\n\nIntuitively, this makes sense. As X grows, there are more (a higher percentage of) poor students in a district. So, as we go from left to right on the horizontal (x) axis, test scores drop.\n\nb = The average of the y's minus (a (from above) times the average of the xs)\n\nb=y-ax 1119.1 - (-561.34)(0.419) = 1354.3\n\nSo, the equation of the line, y ax + b, is:\n\ny = -561.34x + 1354.3\n\nPlug in the x's from the table above and the y's that are yielded in this equation will be the predicted scores.\n\nNow, we need to calculate the correlation coefficient (r) and, more importantly, the coefficient of determination (r2).\n\nWe already have all of the needed components, so we plug them in:\n\n-453.389 T r2 0.8\n\nThat's a coefficient that suggests a strong relationship between income and test scores. It means that 80 percent of the variability in test scores is explained by the variability in income, according to the regression model. The larger the coefficient (up to 1), the stronger the relationship between the independent and dependent variables.\n\nYour regression model still may be off base. It could be totally ridiculous. The formula does as it's told. As you can see from the math, there is no mystical transmutation of the numbers. It's all a matter of averaging, squaring, substracting, etc. - all calculations designed to simply minimize the distance of each actual data point from the estimated line. Keep this in mind in your rage to regress. Don't stop. Just keep this in mind.\n\nNeill A. Borowski can be reached by e-mail at lnborowski@phillynews.com",
  "provenance": {
    "extracted_by": "anthropic/claude-sonnet-4-6",
    "extracted_at": "2026-07-05",
    "verified": true
  },
  "similar": [
    {
      "slug": "1996-april-stats-add-muscle-to-computer-assisted-reporting",
      "score": 0.398
    },
    {
      "slug": "1997-july-using-regression-analysis",
      "score": 0.333
    },
    {
      "slug": "1999-september-returning-to-regress",
      "score": 0.31
    },
    {
      "slug": "2006-july-august-finding-diamonds-in-the-rough",
      "score": 0.278
    },
    {
      "slug": "2000-march-recent-car-stories-2",
      "score": 0.2
    }
  ],
  "html_url": "https://dwillis.github.io/uplink/article/1999-julyaugust-stats-and-such-raging-to-regress.html",
  "json_url": "https://dwillis.github.io/uplink/article/1999-julyaugust-stats-and-such-raging-to-regress.json"
}