Before you ask a school district or state for test score data, think about how you may want to use it. Do your homework. Take the time to look at what other news organizations have done with similar data. Solicit ideas from reporters and editors in your own newsroom and ask them what they'd like to see written. If you have a good relationship with any teachers or members of the local PTA, ask them the same question. Keep things focused. There are so many potentially good stories in these data. Without focus, you risk getting lost, getting frustrated, and producing a lot of mediocre stories instead of a few really good ones.
Demanding details
Try to get student-level data. Some districts keep only minimal information; others gather demographic and socioeconomic information on students and their families. Get as many years of data as you can. Analysis of test scores for a single year is pointless if the focus of your stories is performance. Try to get three to five years at a minimum.
Following is a sample wish list for student-level data. If you can get only school-level data, try to get aggregates like gender and race percentages.
Student ID (Student names are confidential, but sometimes the district or state will generate an ID number for each student that you can use. This will allow you to do longitudinal analysis.)
Grade level, teacher, school attended Gender, race Grade point average Parent education level
Family income information. Participation in free or reduced lunch programs is the most commonly used indicator of poverty or low income. Free lunch participation tends to dwindle as students get older and it becomes more of a social stigma. So, if the district keeps a more stable indicator, like parent's income range, get that, too.
Student ZIP codes or a more specific geographic level, like nodes or census subdistricts. (Student addresses usually are confidential.) If your district has a desegregation plan, chances are it has divided the district into small geographic areas. Even if there is no desegregation plan, the district may divide its area into smaller sections. With any luck, you can match these areas to census and property information to get a feel for the demographics of a student's home environment: Do they come from an area with a lot of rental units or single-parent families? An area where high school or college graduation rates are very low or very high? An area where the average cost of a home is six figures?
Test scores for all subjects tested, plus total battery scores. Ask for NCE scores, as well as percentiles. You can't do math with percentiles. Mobility figures: How often has the student moved within the district? Is there an indicator denoting when the student entered the district from another district? Parent participation numbers. Some schools keep data on what percentage of parents participate in the PTA, school events, report meetings, or teacher conferences.
Statistical differences
School districts will vary widely in terms of the statistical expertise they have on staff. But even the least sophisticated school researchers may assume they know more than you do and that you will mess things up. They may be right. You may mess it up if you don't know what you are doing. Bottom line - know the difference between a percentage and a percentile; know the difference between NCE, scale and percentile scores; and have a good understanding of statistics.
Do you know what this means and how to use it?
y = B + + E
If you don't know what that formula is or how to use it, take a course on linear regression or get yourself a consultant. While you can spend big bucks on consultants, there are less expensive alternatives: another journalist who has done this type of work, someone at your own news organization in quality control or marketing who has a statistics background, or a professor at a local college willing to work for a nominal fee or for free.
Perils and pitfalls
Test score data is full of pitfalls; some may be unique to your data. A thorough knowledge of the school district will help you spot any problems. Things you might encounter:
Universal lunch schools. These are schools where the percentage of low-income students is so high that all students are eligible for free/reduced lunch whether their family income qualifies them or not. This will affect your use of free/reduced lunch as an income indicator. Grade bias. In some districts, certain grade levels traditionally do better on standardized tests than others. In Omaha, it was the 2nd grade. Because of the city's desegregation plan, this biased the results for schools that had only 2nd grade test-takers in their school. If this bias exists, you can choose to exclude these schools or disclose this problem to the readers. Apples-and-oranges demographic comparison. When accumulating demographics to accompany the test score data, make sure you're comparing the same population. For example, if the standardized test is only given to 2nd, 4th and 6th graders, use demographics for the test-taking population only. Means versus medians. If your dataset is small or the scores are not normally distributed, consider using median scores for your analysis instead of mean scores. The more you boil down your dataset (schools, grade levels or even individual classrooms), the more you should consider using medians. Mobility. If you are using mobility as a variable in your regression model, make sure to find out how it is defined. Some schools count only students entering a school or district after the last registration day; some count only students leaving; and some count both types of movement. Inflating prediction. Don't overload your analytic model with predictor variables. If you dump too many variables into a regression model, you will artificially inflate the results. Watch for variables that correlate highly with each other. Putting them both into the model also can cause problems.
Unavoidably incomplete
Remember to acknowledge to readers the limitations of your analysis. You can't create a statistical model that explains all the differences in test scores.
Regression is better at analyzing groups than individual students. You'll find this out yourself if you build a model and apply it to individual student scores and then to aggregate school scores. The model will explain significantly more difference in school level scores than individual ones.
A common criticism of test score analysis is that it "makes excuses" or "sets the bar lower" for poor or minority students. This perspective often comes from people believing that your analysis is predicting how students are going to score on future tests because of their socioeconomic status. While it is true that regression analysis is used for prediction, the model based on historic performance letters of these students or schools (providing you have ample years of data in your analysis). In your stories, you may also consider avoiding words like 'expected,' which for some may connote that you are setting different levels of expectations for different students or groups. Explain how you did your analysis, but expect that you'll never satisfy all your readers. You'll hear from different people if you keep your explanation too simple or vague than if you make the explanation too detailed or complex. Consider providing a moderate amount of detail in the stories and then publishing a more detailed appendix that people can order.
Carol Napolitano can be reached by e-mail Carolnap@owh.com