Numbers Can Be Misleading
Our society likes to boil down assessments of complex systems into short numerical expressions. When we want to know the health of our economy, we look at the Dow Jones average. When checking someone’s health, we start with blood pressure. And when we want to know how our public schools are doing, we look at standardized test scores, which, in California, we derive from the Stanford 9 test.
There is nothing wrong with simple numerical indicators, but it would be foolish to diagnose a complex system from one indicator. The stock market averages were high right up to the 1929 crash, and many illnesses appear in people with normal blood pressure.
Test scores, too, do not by themselves offer a thorough understanding of student intellectual growth. There can be overestimation of student progress or underestimation. The former is apparent in the near-hysterical joy with which we greet score increases of a few percentile points. If an inner-city district’s third-grade reading scores go up two percentile points from the previous year, there is a hue and cry about progress from school staff and from proponents of programs used at the schools.
We need to take into consideration the “Hawthorne effect” when evaluating the meaning of small score increases. In 1924, Elton Mayo did a study of the effects of illumination on productivity at the Western Electric plant in Hawthorne, Ill. The assumption was made that increases in illumination would increase productivity.
Workers were divided into a test group and a control group. Lighting power was increased for the test group, and productivity went up. Lighting was not increased for the control group, but productivity went up for it as well.
Baffled by these results, Mayo and his team tried something new with a group of female workers: They implemented scheduled rest periods, company lunches and shorter workweeks. Productivity, as expected, went up. After a year and a half, the researchers took away everything they had given the women. This was expected to have a negative impact. Instead, their output jumped to an all-time high. Mayo concluded that productivity went up every time he paid attention to workers.
Surely the Hawthorne effect is a factor in educational testing. Merely by virtue of paying attention to students we can expect some sort of increase in test scores. Whether we have permanently impacted a child’s academic life is another question.
The other problem with simplistic reliance on test scores is underestimation of student progress. As an inner-city elementary school teacher for 16 years, I have worked with hundreds of highly intelligent children whose scores were abysmally low on standardized tests. These children were deficient in test-taking skills. We need to find ways to correct their deficiency, but we should not summarily dismiss their intellects in our obsession with test scores.
Standardized tests have given us a frightening picture of the failure of our schools, and we do need a national commitment to raising the scores. But that commitment should be calm, rational and well-informed.
