How many times have you interviewed an "expert" who quickly described a process without going into detail? If you're any kind of interviewer, you made this expert stop and painstakingly explain how the process worked. The process might have been a controversial bond issue, a new communications technology or the treatment for a disease.
Now let's consider the rage to regress. Only at a NICAR session will you hear journalists in the bar late at night discussing how they "ran this independent variable with that dependent variable and came up with an r-square of
Yet, when we use this popular and insightful tool, we never press for an explanation of how the process works. We plug our variables into the black box (SPSS, SAS, Excel, etc.) and out pops the regression results. Many don't seem the least bit interested in how the black box did what it did.
Call me a math nerd, but I like to understand the statistical underpinnings. Obviously you can't hand-crank a regression analysis of thousands of records. But it still makes sense to understand the process that's going on inside the black box.
Our goal in simple linear regression analysis is to come up with a model: + random error. The "y" is your dependent variable (such as test scores). The "X" is your independent variable (such as poverty rates). The "a" is the slope of the regression line and the "b" is the constant (the intercept of the vertical axis). This is the equation of the regression line.
The regression line is fitted from, in essence, many averages. The model tries to plot the line that best fits the data that you've provided. The first step is a scatterplot, simply taking each data point and drawing a point where its X value and y value intersect. The regression line drawn through the points is the best fit, based on the average lengths of the data points from the regression line.
These two charts represent the percentage of students eligible for free lunch in a school (x) and that school's average test score (y). The other columns simply are X-squared, y-squared and X times y.
FREE LUNCH % (x) | x2 | TEST SCORE (y) | y | xy 0.55 | 0.3025 | 900 | 810000 | 495 0.67 | 0.4489 | 900 | 810000 | 603 0.03 | 0.0009 | 1365 | 1863225 | 40.95 0.48 | 0.2304 | 1152 | 1327104 | 552.96 0.12 | 0.0144 | 1218 | 1483524 | 146.16 0.89 | 0.7921 | 856 | 732736 | 761.84 0.76 | 0.5776 | 1010 | 1020100 | 767.6 0.08 | 0.0064 | 1300 | 1690000 | 104 0.26 | 0.0676 | 1193 | 1423249 | 310.18 0.35 | 0.1225 | 1297 | 1682209 | 453.95 SUMS 4.19 | 2.5633 | 11191 | 12842147 | 4235.64
Now the regression math - or "method of least squares" - begins.
Sxx = (The sum of the X squares) - (The sum of the x's, squared/n)
Or, in math notation:
Sxx = - Ex-ex
Substituting from the table above:
Sxx = 0.80769
Now the same for the y's (test scores):
Syy = (The sum of the y squares) - (The sum of the y's, squared/n)
Or, in math notation:
=
Substituting from the table above:
Syy = 12,842,147-11,191°-318,2
Finally, we need a third "average" - the average of the x's times the y's.
Sxy = (The sum of the xs times the ys) - (The sum of the x's times the sums of the y's)/n
Or, in math notation:
=
Substituting from the table above:
=
With these "averages" behind us, we now can find the components of the regression line.
=-561.34
Since a is the slope of the regression line, and because a is negative, the line is negatively sloped - meaning that it goes from the upper left to the lower right.
Intuitively, this makes sense. As X grows, there are more (a higher percentage of) poor students in a district. So, as we go from left to right on the horizontal (x) axis, test scores drop.
b = The average of the y's minus (a (from above) times the average of the xs)
b=y-ax 1119.1 - (-561.34)(0.419) = 1354.3
So, the equation of the line, y ax + b, is:
y = -561.34x + 1354.3
Plug in the x's from the table above and the y's that are yielded in this equation will be the predicted scores.
Now, we need to calculate the correlation coefficient (r) and, more importantly, the coefficient of determination (r2).
We already have all of the needed components, so we plug them in:
-453.389 T r2 0.8
That's a coefficient that suggests a strong relationship between income and test scores. It means that 80 percent of the variability in test scores is explained by the variability in income, according to the regression model. The larger the coefficient (up to 1), the stronger the relationship between the independent and dependent variables.
Your regression model still may be off base. It could be totally ridiculous. The formula does as it's told. As you can see from the math, there is no mystical transmutation of the numbers. It's all a matter of averaging, squaring, substracting, etc. - all calculations designed to simply minimize the distance of each actual data point from the estimated line. Keep this in mind in your rage to regress. Don't stop. Just keep this in mind.
Neill A. Borowski can be reached by e-mail at lnborowski@phillynews.com