{ "id": "1999-september-returning-to-regress", "slug": "1999-september-returning-to-regress", "url": "https://dwillis.github.io/uplink/article/1999-september-returning-to-regress.html", "year": 1999, "month": "September", "monthNum": 9, "headline": "Returning to regress", "kicker": "STATS AND SUCH", "author_name": "Neill A. Borowski", "author_title": "", "affiliation": "", "summary": "This article explains key concepts in regression analysis, including how to handle outliers, interpret t-tests, and make data transformations, while emphasizing the importance of understanding the underlying mathematics rather than relying on computer programs as a 'black box.'", "keywords": [ "regression", "outliers", "analysis" ], "topics": [ "Data & Databases", "Investigative Methods", "Technology" ], "technologies": [ "Excel" ], "full_text": "This is the second of two parts on how to do a regression analysis. This issue's column deals with regression outliers, the t-test and making transformations. In the last issue, the main advice was not to succumb to the \"black box\" syndrome - being awed by the ease of running regression analyses through computer programs with no appreciation for what is going on behind the curtain. The math behind the regression process was explained in detail. The regression model seeks to fit a line through - or close to - all of your data points. In a plot of the data, the x-axis holds the independent variable and the y-axis holds the dependent variable. That line is one that you might recall from high school algebra: y = ax + b + random error. The \"y\" is your dependent variable (such as test scores). The \"x\" is your independent variable (such as poverty rates). The \"a\" is the slope of the line and the \"b\" is the constant (the intercept of the vertical axis). This is the equation of the regression line. You may have seen the equation stated in different ways. Some statistics texts prefer the notation of y = bx + a. Ory=mx+ b. Or, even fancier, the equation y + Blx E Select whichever you're most comfortable with. The goal in regression is to show how much the dependent variable is related to the independent variable. This quest to find the truly predictive model is one undertaken by social scientists, physicists, epidemiologists, quality-control managers and others in nearly every field. If you're interested in practicing regression, there are plenty of free, downloadable datasets available on the Internet. Two great sites are the Journal of Statistics Education Data Archive at www.amstat.org/ publications/jselarchive.htm and the Data and Story Library at http://lib.stat.cmu.edu/DASL/, where you'll even find datasets covering acorns and oak trees and passenger car mileage. In most cases, the size of the dataset is small, making regression with MS Excel and other spreadsheet programs relatively easy. Here are some points to remember in the rush to regress. Dealing with outliers An outlier is a data point (observation, record, etc.) that represents an extreme value when compared with other values in the data. Outliers can be gremlins in all data analysis, whether you're calculating the mean or running a complex regression. If you're going to be \"hanging your hat\" on your regression results understand the t and the tests. If you don't pay attention to these tests, your whole hypothesis may be rejected without your knowledge. The simple way of thinking about an outlier is in terms of its effect on the average. Consider a study of speed in fatal automobile accidents. The estimated speeds of the cars involved in such accidents on one stretch of road in one year are 65, 55, 95, 60 and 65. The average speed is 68. Yet, all but one speed is less than 68. The 95-miles-per-hour speed skews the data. The average without the extreme value is 61 - certainly more representative of the group. Extreme values or outliers also can affect the outcome of your regression analysis. Many of us have seen the outliers in a scatterplot of test scores, with the percent of students eligible for free lunch as the independent variable. We see relatively poor schools with unusually high grades (the \"overachievers\") and relatively wealthy schools with unusually low grades (the \"underachievers\"). In these cases, the news may be in the outliers. What secrets might you find in the poor schools that do well on tests? However, a common practice in regression analysis is to throw out the outliers to try to achieve a higher coefficient of determination (r2). With the outliers included, maybe your r2 is 0.53, meaning that 53 percent of the variability in test scores is explained or predicted by the variability in income. By eliminating the outliers, your r2 may increase by several points - making you feel even more confident about a relationship. How far off are some data points? In SPSS, SAS and EXCEL, for example, you can save and study the residuals, which are the differences between the actual value of each observation and the predicted value from your regression model. You also can calculate the residuals by hand by substituting each \"x\" into the regression equation to solve for \"y\" (y = ax + b). Then find the difference between the actual \"y\" (for example, the test score) and the predicted \"y\" from the regression equation. (You can visualize residuals in a scatterplot of your data by drawing vertical lines from the datapoints to the regression line. The lengths of the vertical lines represent the residuals.) Watching the t-test The t test and the F test are part of the regression output in statistical programs and Excel. Yet, many people may pay no attention to these tests in their zeal to achieve the highest-possible r2. Pay attention to these tests, which report both the t and the F values, as well as their significance levels. These tests will indicate if there is a serious problem with the regression model. The ideal situation is to see \".000\" or something close to zero in the significance box. If you see much more than zero, this is cause for concern. If you're going to be \"hanging your hat\" on your regression results, spend some time with statistics texts to understand the t and the F tests. They are not difficult to understand, but they are involved. If you don't pay attention to these tests, your whole hypothesis may be rejected without your knowledge. Making transformations If you plot the data and find there are curves or bends in the fitted line, you may want to express the data differently, or \"transform\" it. I remember discussing this process with a colleague at an IRE and NICAR seminar and he gasped at the thought of changing the data. However, transformations are uniform and change all of the data by the same amount-much like multiplying both sides of an algebra equation by 2 to get rid of the 1/2 on one side of the equation. Depending on the severity and direction of the curve in the fitted line, you may want to transform either the independent (\"x\") or the dependent (\"y\") variable. Methods include substituting the square or square root of \"x\" or \"y\" or taking the natural logarithm of one of the variables. The goal is a better fit - a higher r2 and a straighter regression line. Clearly, there is a lot to know about regression analysis. It takes more than simply running a regression using SPSS or Excel and relying on each program's online Help facility to explain what you are doing. Pick up a statistics text or two - particularly those used in graduate school. Many college statistics departments have put up Web sites that explain the basics. Some actual statistics classes are online and available to the public. A good place to start is in Yahoo at http://dir.yahoo.com/ SciencelMathematics/Statisticsl But, most important of all, know that there are a lot of gears whirring in the black box and that it's worthwhile understanding a bit about how they mesh (or fail to mesh). Neill Borowski can be reached by e- mail at nborowski@phillynews.com", "provenance": { "extracted_by": "pre-v2", "extracted_at": "", "verified": true }, "similar": [ { "slug": "1999-julyaugust-stats-and-such-raging-to-regress", "score": 0.31 }, { "slug": "1996-april-stats-add-muscle-to-computer-assisted-reporting", "score": 0.218 }, { "slug": "1997-july-using-regression-analysis", "score": 0.203 }, { "slug": "2006-july-august-finding-diamonds-in-the-rough", "score": 0.173 }, { "slug": "1997-april-getting-back-to-basics", "score": 0.154 } ], "html_url": "https://dwillis.github.io/uplink/article/1999-september-returning-to-regress.html", "json_url": "https://dwillis.github.io/uplink/article/1999-september-returning-to-regress.json" }