Wine judges are rather unsteady, study finds

Judges at the California State Fair wine competition scored poorly at giving the same wine an identical rating when they tasted it multiple times in a blind tasting.

That was the conclusion of a four-year study of judging decisions at the California State Fair Wine Competition by retired Humboldt State professor Robert Hodgson.

“Consumers should have a healthy skepticism about the medals awarded to wines from the various competitions,” he said.

Hodgson’s findings have prompted state fair officials to consider making changes in the way they operate future wine competitions.

In a study published Wednesday by the Journal of Wine Economics, Hodgson wrote that only 10% of the judges were able to consistently give the same rating, or something very close, to the identical wine sampled multiple times in a large blind tasting.

At the opposite end, another 10% of the judges gave the same wine far different ratings, ranging from worthy of a gold medal to deserving of no medal at all on successive tastings. The remaining 80% of the judges also varied in their ratings, but by a narrower range.

State fair officials, who cooperated with the study, said Hodgson’s findings provided valuable information that could be used to improve wine competitions. The next state wine judging is set for June 10-12.

“We want to do whatever we can to smell and taste wines to get the best results,” said G.M. “Pooch” Pucilowski, a professional wine educator and chief judge for the state fair.

The competition plans to reduce the number of wines sampled per day -- which can approach 150 or more -- to as few as 75 to help judges avoid the sensory fatigue that can cloud their ratings.

The fair is also using the data “to weed out” judges who year after year are unable to rate different samples of an identical wine consistently, Pucilowski said.

Finding ways to evaluate the skills and consistency of judges is an important issue for wine competitions, which often draw from the same small pool of industry members and aficionados for their rating panels. It’s not unusual for judges to work as many as six different competitions annually.

Wine shoppers such as Glendora urologist Dr. Charles Metzger rarely look for wines that are medal winners when deciding what to purchase.

“I mostly look to what other people recommend, and then I look at the 100-point ratings from publications like the Wine Spectator,” Metzger said.

“I found that some of the Los Angeles County Fair winners that I tried were just not that good,” he said.

But Metzger’s wife, Barbara, has had good luck buying wines that have won awards at competitions, he said.

“The medal winners tend to be less expensive than the $40, $60, $80 wines you see in Wine Spectator,” Metzger said.

Hodgson said he doesn’t put any more trust in the 100-point-scale ratings of wines from magazines and newsletters than he does in medal winners.

“Consumers need to gain more self-confidence in their own opinions and tastes rather than listen to what other people think wine should be like,” Hodgson said.

Last year, 649 wineries entered 2,917 California wines into the annual state fair contest, the oldest wine competition in America. More than half, or 1,587 wines, won awards.

Hodgson, an oceanography professor who also taught statistics at the university, owns the small Fieldbrook Winery just north of Eureka, in Humboldt County. He said he designed the study because he didn’t understand why “we would have wines that we sent off and would get gold medals in some competitions and in others would get poop. It seemed like a gold medal was just a matter of luck.”

Hodgson said he learned that the judging business is characterized by inconsistent decision-making by judges and wide variability among competitions.

“Wine judges in the setting of a competition must make about a hundred decisions a day. It is in this environment where I think their ability is taxed beyond a reasonable level,” Hodgson said.

He also discovered that a “super judge” who is consistent in his or her ratings one year does not maintain that superiority the next year.

“This emphasizes the chance argument in placing awards,” Hodgson said.

The findings of the study create a conundrum for Hodgson and his small winery.

“I use gold medals to sell my wine,” he said, “and now I have written this paper saying the wine competition system that awards those medals isn’t perfect.”

--

jerry.hirsch@latimes.com