Should a School’s Quality Be Judged by Test Scores?

As California Assessment Program scores have gained statewide prominence, they have stirred passionate debate among educators, some of whom believe the test unfairly lumps all the complexities of education into a bite-sized, shallow set of numbers.

“My personal belief is that CAP scores are a very useful litmus test,” said Carol Berg, assistant superintendent for instruction, curriculum and programs at Newport-Mesa Unified School District. “But it’s totally inappropriate to use them to judge a school’s whole program, and I shudder to think that when it comes to choices, people are judging schools based on their CAP scores.”

But judge them they do.

Bill Honig, state superintendent of public instruction, has made CAP scores a cornerstone of his drive for steady educational progress and accountability. Orange County educators often hold up the scores as evidence of the county’s academic excellence.

Area realtors, who tout the county’s educational reputation as a way of luring home buyers, also make use of CAP scores. Realtors will often provide prospective buyers with the scores to demonstrate a local school’s excellence. Some even draw up maps showing the CAP scores achieved by schools serving certain areas.

Chuck Colesworthy, owner-broker of Century 21 Coastland Realty in Newport Beach, said he takes advantage of his office’s proximity to Andersen Elementary School, a Newport-Mesa school that topped all other county CAP scores in sixth-grade reading.

“You better believe I point to it. I use it all the time,” Colesworthy said. CAP scores “absolutely” have an effect on potential home buyers, he maintained.

That causes mild concern by some civil libertarians, however. In addition to telling something about a neighborhood’s educational achievement, CAP scores are related to a community’s affluence and racial makeup. Directing home buyers toward high-scoring communities, thus, could have the effect of steering them to predominantly white neighborhoods.

“My eyebrows are a little raised,” said Linda Burstyn, communications director of the Southern California American Civil Liberties Union.

Burstyn added that she was not aware of complaints about the practice and said the ACLU has not researched it in depth.

The popularity of the CAP scores as achievement yardsticks for schools has grown in recent years, mostly as a result of Honig’s efforts. The superintendent releases results with great fanfare, and educators have responded, gearing curriculum to the tests in order to avoid the public embarrassment that comes with low rankings.

“If you’re going to be embarrassed by low test scores, you’re going to intuitively respond to the test by making changes in your curriculum,” said James Cox, director of research and development for the Anaheim Union High School District. The CAP tests question students on elements of the state’s curriculum, so teaching inside that framework is encouraged.

In interviews with more than two dozen Orange County teachers and education officials last week, all endorsed the curriculum guides and agreed that the CAP tests help enforce the state’s efforts to get schools to adopt them. But CAP scores frequently are interpreted as providing an overall judgment on a school’s performance, and that, critics say, is dangerous.

“They’ve gotten blown out of proportion,” said Berg, whose Newport-Mesa district ranked well above state and county averages in nearly all areas tested at all grade levels. “A school’s success is being measured entirely by how well certain students perform on these tests, and that’s just not correct.”

Cox strongly supports CAP but says it is important to treat the annual results with caution. For while the tests measure a school’s program, he said, they also gather in volumes of extraneous data that can skew scores.

Cox, whom many educators consider one of Orange County’s foremost CAP experts, said the test measures at least three variables in addition to the school’s program: demographics, testing conditions and the degree to which a school’s curriculum matches the test questions.

“What happens is that when a score goes up or a score goes down, the inference is that something is going on with the program,” Cox said. “That’s not always the case. Often something is going on with one of the other factors.”

In particular, Orange County schools often suffer from changing demographics. Schools can drop substantially in their CAP scores without changing their program at all, simply because they receive a large influx of students who do not speak English well.

CAP also uses a complicated brand of analysis, known as “matrix sampling,” that some educators complain makes it possible for whole schools to suffer as the result of very few lagging students. Using matrix sampling, dozens of different tests will be given to a classroom of students, who will each complete a section of the matrix.

The result, Cox and others said, is that the test more accurately portrays the ability of a group of students when that group is very large. In small schools, a few low scores can dramatically affect the entire group.

The matrix sample discourages cheating because students sitting next to each other will usually not have the same test.

Still, even critics of CAP tests acknowledge that the scores gives educators and parents an important tool for holding schools accountable for progress. And even in districts that have historically posted low scores, the test has supporters, who say that it helps schools identify problem areas and begin the complicated task of addressing them.

“We’re very supportive of CAP,” said Vergil Hettick, director of research and evaluation for Santa Ana Unified School District. “It helps us target schools that need help. Some of ours do, and CAP is making it easier.”