Problems Undercut Debut of CLAS Tests, Panel Says : Education: Experts say outside firm should administer the exams. But they praise the program for innovation.

Budget constraints, scoring glitches and the difficulties inherent in such a complex new testing system have undercut the pioneering California Learning Assessment System exams, a panel reviewing the CLAS tests’ first rocky year has concluded.

Most individual school results for the 1993 exams were less reliable than promised, and some 250 schools statewide received scores that may be woefully inaccurate because of severe processing problems, the panel found in its 62-page report, to be released today.

The three-member panel of statistical experts recommended that the state Department of Education hire an outside firm to administer the tests. The group also called for the department to turn over responsibility for scoring and interpretation of the results to a statistics expert and to hire a full-time technical coordinator to oversee the effort.

The panel also cautioned against releasing individual student scores, scheduled to begin this year, until the bugs are worked out and parents and teachers are thoroughly briefed on how to interpret the results. The governor’s office has been pushing for individual results so parents and teachers can gauge students’ progress.

The exams, intended to pinpoint a school’s academic strengths and weaknesses by assessing how well students can think and write about what they have learned, were given for the first time last year to about 1 million public school students in fourth, eighth and 10th grades. To save money, the department decided to score only some of the exams in reading, writing and mathematics before releasing results for each school.

Almost immediately, questions surfaced about the accuracy of the scores, which were widely published and used by parents and others to evaluate schools. A Times computer analysis published in April showed that thousands of schools received shaky results, and that at least 150 schools had statistically invalid samples of student scores.

At one San Bernardino County middle school, for example, only 1% of the math tests were ever counted. Two schools--Vicentia Elementary in Corona and Columbia Elementary in Tuolumne County--received results based on exactly one student’s work. Dozens of others across the state had only a handful of tests scored though scores, or even hundreds, of students participated in the exam.

The panel of three statistical experts, appointed by the department in April to review its scoring procedures, found that there were serious shortfalls in the numbers of 1993 exams graded in about 250, or 3%, of schools. In those cases, additional exams were scored and results adjusted, the Department of Education said.

But the experts, headed by Lee J. Cronbach of Stanford University, concluded that “few school-level reports in 1993 had adequate reliability.”

Problems inherent in the sampling procedures were compounded by lost test booklets, errors matching student identification sheets to the booklets, mistakes by the contractors helping administer and score the tests, and the rush to get scoring done and results released before the 1994 round of testing began in the spring.

“Inadequate review under time pressure led to the release of some reports that were manifestly untrustworthy,” Cronbach added.

To make matters worse, many parents and school staffs had expected all the tests to be graded, the report noted.

Although the panel was assigned to look primarily at the sampling procedures, it also noted difficulties in scoring the open-ended, or “performance-based” questions that are the hallmark of CLAS and other pioneering testing systems around the country. Instead of merely selecting the correct answer from a list of possibilities, students are being asked to solve math problems and explain their methods or read a passage from literature and write about their responses.

But the panel commended CLAS officials for some of their work.

“CLAS and its contractors have embarked on an unprecedented and imaginative project . . . and they have done many things well,” Cronbach wrote, adding the 1993 efforts were “highly successful as a trial” and shed valuable light on many aspects of the just-emerging field of performance-based assessment.

“All the shortcomings of CLAS 1993 can be remedied,” Cronbach added.

Acting state Supt. of Public Instruction William D. Dawson said he welcomed the report and has already acted on some of its recommendations. Scoring of this year’s tests, done by specially trained teachers at sites around the state, already is underway and reflects the panel’s suggestions, Dawson said.

The report comes as CLAS, which is also under attack from religious conservatives for the contents of its reading and writing tests, faces an uncertain future. Next week, legislators will work on a compromise bill that would continue the system--renamed the California Comprehensive Testing Program in state budget sessions this summer--while addressing many of the criticisms.

Gov. Pete Wilson’s education adviser, Maureen DiMarco, who has been critical of how the testing has been developed and carried out, said the report “not only validates our concerns and the concerns of others but also gives us even greater impetus to continue to work to revise the assessment system.”

The governor’s insistence on individual scores could be met by adding more traditional, skills-measuring methods, such as multiple-choice questions, she added, noting some of the revisions sought by her boss and other CLAS critics already have been added to the proposed legislation.

Times staff writer Jodi Wilgoren contributed to this report.