Data Foul-Ups Delay Analysis of School Tests

Major foul-ups in the handling of score sheets are hampering efforts to analyze results from the most ambitious student testing program in California history, state officials say.

The worst problems affect as much as 18% of the test data statewide and could sharply limit the value of a program that cost taxpayers about $35 million and was intended to give them a public school report card.

So far, the public has not yet learned how students statewide from various ethnic groups performed on the Stanford 9 tests of basic skills, how girls did compared to boys, and how students fluent in English did compared to their peers who speak limited English.

Nor is there any readout on the performance of second- and third-graders who were put into classrooms limited to 20 students each.

State officials, led by Gov. Pete Wilson, frequently cited the need to review the 2-year-old class-size-reduction program in their push last year to resume standardized testing.

Originally, reports on those topics and others had been expected this summer, in time for educators and policymakers to use the information before most children return to school in September.

But now that looks unlikely. Officials from the state Department of Education and the State Board of Education and the testing company are trying to iron out the difficulties; they have set no deadline for when the statewide data will get cleaned up.

Money is at issue. The company, Harcourt Brace Educational Measurement, is demanding additional payments for many of the fixes. Those could exceed $250,000. The State Board of Education, meanwhile, holds a $1.7-million bond that Harcourt Brace could forfeit if the state concludes that the company has not meet its obligations.

Another issue is who’s to blame. The most significant foul-ups involved a failure to obtain complete demographic data from student score sheets. The testing company says local school officials did not follow directions. But many educators say the directions were unclear.

Education Experts Are Disappointed

The delay disappoints education experts who have been hoping to arrive at deeper conclusions about public schools than simply whether students as a whole are above or below the national average.

“One of the reasons you have the test scores is to look at individual effects,” said Richard S. Brown, a project director at UCLA’s Center for the Study of Evaluation. Not being able to do such analysis, he said, “diminishes the value of the testing system.”

The Standardized Testing and Reporting program, known by the acronym STAR, is the first statewide public school testing effort in four years and the first since the 1960s to provide individual student scores.

More than 4.1 million students from grades 2 to 11 took the multiple-choice Stanford 9 tests last spring in reading, mathematics and written expression. Elementary and middle school students also were tested in spelling and high school students in science and social sciences.

Overall results from the program posted in July showed the state’s students below or hovering near the national average. The results also showed, not surprisingly, that students classified as “limited English proficient” did far worse on tests given only in English.

Although those figures may have satisfied the initial public demand for a reckoning of how the schools are doing, they barely tapped the information the testing program was meant to yield.

Every score sheet was supposed to identify a test-taker’s ethnicity, gender, home language, length of time spent in a given school, degree of English fluency, level of parent education and participation in special programs such as class-size reduction or classes for “gifted and talented” students.

But about 18% of the testing records, accounting for 750,000 students, were missing some or all of that valuable demographic data. For that reason, said Gerry Shelton, a testing administrator for the California Department of Education, follow-up reports are on hold.

State and Harcourt Brace officials agree on the mechanics of what happened. Many school districts chose to pre-label score sheets with student demographic profiles using bar codes, figuring that would be more accurate than filling out questionnaires by hand.

But Harcourt Brace programmed its computers to ignore that pre-coded information if students or teachers penciled in answers to any of the demographic questions themselves. The pre-coded information disappeared even if a student only filled in the bubbles corresponding to his or her name.

Harcourt Brace says it will fix the problem for any district willing to pay for it, and to forward the corrected information to the state. Its fee is 35 cents per student, or less if districts provide a complete new demographic data record.

“We feel we have done everything as well as we can,” said Dave Osberg, a senior program director for Harcourt Brace. “We just can’t give away the company here to solve all the problems--if they were district-created.”

Osberg said the company hopes to give the state corrected data by early fall. He could not guarantee that all the missing data would be found.

Osberg said the company plans to fix, at no charge, another significant problem in which scores for several hundred schools were mislabeled. In some cases, students in special programs--such as classes for the gifted--were mistakenly grouped into “schools” that are not recognized by the state. The groupings skewed the scores for the real schools to which they belonged.

Declining to “point blame” for the missing demographic data, Shelton said state officials still hope to reach an accord. But Supt. of Public Instruction Delaine Eastin said the state’s leverage is limited--despite the performance bond--because the testing company contracted with each of the state’s 1,000 school districts rather than the state itself.

The State Board of Education is expected to discuss the STAR program next month. In July the board gave school districts permission to begin paying the $17 million owed Harcourt Brace. It also reaffirmed its power to hold the company accountable through the performance bond and asked state and company officials to work out the glitches.

If that fails, state officials will consider producing reports by sampling those student records that are complete.

The impact of the foul-ups on the Los Angeles Unified School District appeared to be less severe than on school systems elsewhere, perhaps because the district has used the Stanford 9 for two years. A district official said that only about 6% of test-takers in the district appeared to have missing English-language proficiency data.

But in the Elk Grove Unified School District in Sacramento County, school officials estimate that data were missing for about one in four students tested.

Elk Grove Supt. Dave Gordon said that he doubted that school districts were at fault but that his district would probably pay the company to plug the gaps--reluctantly.

“As a matter of customer service, and service to kids and parents in the state, we think the testing company ought to make it right,” Gordon said.

Times education writer Richard Lee Colvin contributed to this story.