Study backs ‘value-added’ analysis of teacher effectiveness

Teachers’ effectiveness can be reliably estimated by gauging their students’ progress on standardized tests, according to the preliminary findings of a large-scale study released Friday by leading education researchers.

The study, funded by the the Bill and Melinda Gates Foundation, provides some of the strongest evidence to date of the validity of “value-added” analysis, whose accuracy has been hotly contested by teachers unions and some education experts who question the use of test scores to evaluate teachers.

The approach estimates a teacher’s impact by comparing his or her students’ performance on standardized tests to that in previous years. It has been adopted around the country in districts including New York City, Washington, Houston and soon, if local officials have their way, in Los Angeles.

READ THE DOCUMENT: The full text of the Gates Foundation report, annotated by Times Staff Writer Jason Felch

The $45-million Measures of Effective Teaching study is a groundbreaking effort to identify reliable gauges of teacher performance through an intensive look at 3,000 teachers in cities throughout the country. Ultimately, it will examine multiple approaches, including using sophisticated observation tools and teachers’ assessments of their own performance.

The results have been eagerly awaited by education officials and come amid a national effort to reinvent teacher evaluations, which for decades have been based on occasional, cursory observations by principals who give passing grades to the vast majority of teachers.

But some experts and teachers union officials believe the findings are premature.

“The hope of [the project] was to figure out the kind of instructional practices that would help improve student achievement over time, but this preliminary report does not do that,” said American Federation of Teachers President Randi Weingarten, who has collaborated closely with the researchers. “We’re disappointed that it was rushed out when even the authors admit it is incomplete.”

The preliminary report released Friday focuses on two measures of teacher performance: value-added analysis and student surveys.

Both tend to identify the same teachers as either effective or ineffective, the study found, suggesting they could be used in tandem to give teachers better feedback about their performance. In both cases, findings on which teachers were effective held up when those teachers taught a different class of students, the study found.

The study also found that student gains on standardized tests reflected meaningful learning and critical thinking skills, not just test preparation or memorization -- a frequent concern of critics of the value-added approach.

Because value-added measures were so reliable at predicting teachers’ future performance, the researchers urged school districts to use it as a “benchmark” for studying the effect of other measures.

“The evidence on student achievement gains is like a giant divining rod,” said Thomas Kane, a professor of education at Harvard University and director of the research project, in an interview. “It says, dig here if you want to learn what great teaching looks like.”

A growing body of evidence suggests that there are dramatic differences in teacher effectiveness that are not reflected in the subjective evaluations now in place.

The Times began publishing stories in August using value-added analysis to estimate the effectiveness of thousands of district teachers in raising test scores. Drawing on data the district had largely ignored, The Times found huge variation among teachers with similar students. In an online database, the newspaper released the ratings for 6,000 elementary school teachers and plans to release middle school ratings in coming weeks.

The stories and database fueled an intensive debate nationally over how teachers should be evaluated and whether the results should be made public.

Teachers unions and some education experts have argued that value-added is an unreliable measure that encourages teaching to the test.

The study released Friday addresses several of those concerns, Kane said.

“There are folks out there worry that value-added is measuring a statistical aberration,” Kane said. “If that were true, you wouldn’t see any correlation to other measures.”

For example, value-added was found to be a reliable predictor of future performance on sophisticated tests that measure higher-level concepts, particularly in math.

“Teachers who are producing gains on the state tests are generally also promoting deeper conceptual understanding among their students,” the researchers found.

Teachers whose students said they “taught to the test” were, on average, lower performers on value-added measures than their peers, not higher, the study found.

The researchers, who include experts from Stanford University, Dartmouth College, RAND Corp. and the Educational Testing Service, also acknowledged limits to the value-added approach. Scores were not available for all teachers and the estimates were often volatile from one year to the next, though they still predicted teachers’ future performance well. Because they are based on a narrow measure – standardized test scores – and don’t provide teachers with feedback on how to improve, they should be accompanied by other performance measures, the authors said.

The study also found that feedback from students as young as fourth-graders was a reliable predictor of teacher performance, particularly when they were asked about a teacher’s ability to manage a classroom and challenge students with rigorous work.

Student surveys are common in higher education but rarely used in elementary and secondary schools.

Jesse Rothstein, a professor of economics at UC Berkeley who has been critical of the value-added approach, says the preliminary results echoed other research in the field but didn’t answer some of his key concerns, such as how results are affected by the way students are assigned to teachers.

“The good stuff isn’t done yet,” he said.

In the study’s second year, teachers will be randomly assigned to new classrooms , eliminating any potential bias caused by how students are placed with teachers. Results from those analyses will be released next spring, with a final report expected in 2012.

jason.felch@latimes.com