Lake Wobegon effect — Uplink Archive

The Tennessean investigated Tennessee's claim of above-average student test scores and found significant discrepancies between norm-referenced tests and independent assessments like NAEP and ACT, revealing the state's scores were inflated. The investigation discovered that 80% of states using the same TerraNova test showed above-average results, but only 50% scored above average on the NAEP test, illustrating how test selection and presentation can distort educational performance claims.

Each fall, the state of Tennessee releases its school test scores and every year The Tennessean dutifully runs three open pages of scores and graphs showing how local schools are doing. One perennial result is that Tennessee's students score above the national average in most subject areas, a fact trumpeted by the state department of education.

Last fall, we decided to look into that claim. What resulted was a quick one-week story that called into question the basis of this claim. We also learned an important lesson on how information is presented and the importance of questioning basic assumptions.

At the heart of the matter is the way standardized tests are done nationwide. State assessment programs often include two elements: a norm-referenced test and a criterion referenced test. The first allows the state to look at how the kids in the state match up to national standards. The second is how kids match up to state standards. It was the norm-referenced test we were interested in. Tennessee students ranked between the 49th and the 59th national percentile rank depending on the grade level and the subject area of the test. To check whether these numbers were inflated, we asked around about other tests that might corroborate these results. This is always a good technique to check the results of any study or statistic. What we found was the National Assessment of Educational Progress or NAEP test. The NAEP test, which is administered by the Department of Education, is given in one subject area every couple of years to a sample of students across the country. In 1998, students were tested in reading and in 1996 and 1992 students were tested in math. The comparison surprised us. Tennessee showed up below the national average in both tests. The discrepancies between the two tests were large. We checked another test. We found Tennessee students scored below the national average in every subject area of the American College Test or ACT given to high school students.

Checking inflation

To verify the extent of the inflation, The Tennessean built a spreadsheet of all states that used the same test we did, the TerraNova published by CTB/McGraw Hill. We found 17 states that used the TerraNova but only 11 states used it to calculate national norms. We then got the national percentile rank for all of those states. In 80 percent of them, the median student in a particular grade level and subject area was above the national average. We then used the NAEP for comparison and found of those same states, only 50 percent of the median student was above average.

There were limitations of our methodology (i.e. states tested at different grade levels, they used different forms of the TerraNova, NAEP years didn't correspond to some TerraNova years). To lessen these effects, we aggregated the data and didn't try to make too specific of a comparison.

The discrepancies didn't surprise testing experts. In fact, they expected them. When we asked why, the answer was uniform: the Lake Wobegon Effect, named after the mythical town in Minnesota Public Radio's "A Prairie Home Companion" where "all the children are above average".

The Lake Wobegon Effect is an inflation of the national percentile rank due to discrepancies between the current test scores and an outdated, faulty or limited national average. All our states soon report their students are above average.

Most national norm-referenced standardized tests are normed every seven or eight years. The norming process involved testing a stratified sample of hundreds of thousands of students across the country to derive the national average. The norming process is rigorous, expensive and very time consuming for test publishers.

Limitations

But the process has some serious limitations.

Age: If test scores rise over time like they have over the last decade, more and more kids suddenly appear above average.

Teaching the test: States that use the same or similar tests over time show a rapid rise in test scores. When those tests change, the scores drop dramatically. Teachers learn what is on the test and may teach types of problems or even specific problems.

Environment: Students who help norm a standardized test take the test under a vastly different environment than students who are taking it later as part of an operational assessment program. In a norming test, students are completely unfamiliar with the type of test. Imagine taking the SAT having never seen the types of questions on it. In an operational environment, students may have been taking the same test for years and may have a much higher comfort level. Also, in an operational environment, there is enormous pressure on students and teachers for the scores to be high whereas students in the norming process know that how well they do doesn't matter to them. To test this effect, the University of Iowa re-normed a norm-referenced test in an operational environment and found large differences in what the national average would be.

Cheating: It's more common than people think and can skew results upward.

While the Lake Wobegon Effect is old, it continues to skew state results, sometimes very seriously. Be aware that every norm-referenced test is a victim of this effect. The result in your state may be an apparent upward trend in scores that just isn't real.

Questions to ask

Here are some questions you should ask your state assessment officers:

Ask your state's test authorities how they calculate the national average, what test they use and when it was last normed. It's a complicated picture because states may use an off-the-shelf standardized test, part of a test, or just some test questions from a testing company. These questions are often mixed with criterion-referenced test questions.

Ask how often your state changes test forms and how many questions are repeated every year.

Ask about cases of cheating. Often these are called 'anomalies' or some other innocuous term. The state testing office collects the information because they must remove these cases from their statistical analyses. Often it may be a matter of a teacher giving students too much or too little time, but sometimes it is far more egregious.

Compare your schools test results to the NAEP, ACT, SAT or other standardized tests. Also look at trends. If your students are improving on a state test and not on NAEP, you could have a problem.

Talk to state assessment coordinators in other states. They often know what manipulations other states use to make themselves look good.

Find out how your state deals with students who need special accommodations for the test.

Drew Sullivan can be reached by e-mail at drew@tennessean.com