Behind the Ratings

My students and I tasted our way through a half dozen Australian white wines. We were largely in agreement until we came to the 1996 Rosemount Hunter Valley “Show Reserve” Semillon, which I judged to be the finest wine of the night. “It tastes so odd,” they protested.

“Think orange marmalade and lime,” I suggested. “Notice the rich texture. This wine’s a classic, and it will be even better in five years.”

Although these students were far from novices, none of them was familiar with Australian Semillon, and they therefore judged the wine unfairly. Or did they? Is the quality of a wine conditioned by its type, or is wine quality an absolute? What constitutes wine quality, anyway?

In fact, the determination of wine quality is an inexact discipline whose standards vary according to the practitioner, and sometimes according to the type of wine. Many wine critics judge wines against a mental yardstick of greatness developed over years of wide-ranging tasting, while future winemakers who study at UC Davis learn that the definition of wine quality must also take into consideration a wine’s price.

An unusual voice entered into the discussion of wine quality in October when Consumer Reports magazine, better known for its ratings of automobiles and small appliances, published its first wine report in 26 years. The magazine evaluated 57 wines and named four of them, ranging in price from $4 to $11, as the finest in their respective categories. The “best wines”--Beringer Napa Valley Chardonnay, Beringer California White Zinfandel, Napa Ridge Central Coast Cabernet Sauvignon and Walnut Crest Rapel Valley (Chile) Merlot--were not startling choices, considering that the evaluation focused on widely available wines costing under $15 from those four grape varieties. But some of the magazine’s assumptions illustrate just how arbitrary the criteria of wine quality can be.

Two anonymous “industry authorities” rated the 57 wines simply on flavor complexity and absence of off-notes. These are understandably lenient criteria, since they were evaluating inexpensive mass-market wines. On the same basis, it also made sense for them to give lower marks to wines that were not immediately drinkable, such as Cabernets from Robert Mondavi Winery and Stag’s Leap Wine Cellars that would need aging to soften their tannins.

However, the writers of the report were apparently oblivious to the skew of these principles when they proclaimed, “Many outstanding wines cost less than $10 a bottle.” All the tasting showed was that some of the best wines in the under-$15 category cost less than $10!

The problem in Consumer Reports’ approach to wine is summed up by the fact that it referred to “testing”--rather than “tasting”--the wines. There is a vast difference between testing, which implies objective, scientific evaluation, and tasting, a subjective and experiential judgment.

“I don’t believe you can absolutely test quality in wine, except the way that wineries do, by measuring a wine’s alcohol or phenolics,” says Jim Gordon, managing editor of Wine Spectator, a biweekly magazine whose ratings of wine are among the most influential in the country. “What we do is taste for quality, which is subjective in nature.”

Most professional wine tasters use criteria such as balance, length on the palate, complexity, flavor concentration and trueness to the wine’s type. Judges downgrade wines that have what they perceive to be flaws, though today, when the great majority of wines are sound and grossly flawed wines are rare, the notion of what constitutes a flaw is increasingly a matter of opinion.

The criteria may sound technical, but assessing them is inherently subjective. “You just can’t measure wine’s quality mathematically the way you can measure how much dirt a vacuum cleaner picks up,” says Gordon.

Even a bastion of scientific winemaking such as UC Davis acknowledges that wine quality is subjective. “We don’t have a yardstick of quality because quality is such a variable measure,” says Carole Meredith, professor of Viticulture and Enology at Davis. “The one thing you can say about wine quality across the board is that you must consider whether the wine has any defects. But while some defects, such as corkiness, are obviously a flaw for everyone, other characteristics, such as high tannin, are acceptable or not depending on the tastes of the consumer who is purchasing the wine.”

Ironically, considering how subjective all this is, wine quality is usually expressed in a number score. The most widespread scoring system, popularized by Robert M. Parker in his newsletter, the Wine Advocate, gives 100 points to a wine of the highest possible quality.

The apparent precision of ratings like 89 or 91 can lead wine buyers to conclude that a wine’s quality is objectively measurable in tiny increments. But of course the points are assigned subjectively, rather than through any rational, documentable process. There are wine rating systems that provide tasters with a breakdown of characteristics and a range of points attributable to each, but many tasters prefer to weigh the overall characteristics of a wine and assign a number that represents the wine’s quality on a scale ranging from undrinkable to ideal.

But not all subjective evaluations are equal. Tasters who lack broad experience of wines from all over the world or a specific understanding of the type of wine under consideration can have a warped idea of the quality of a particular wine. As my students did with the Australian Semillon.

When greatness is one end of a taster’s quality scale, some types of wine are simply destined to run in the middle of the pack. Charles Laverick, tasting director of Beverage Testing Institute in Chicago, which produces ratings and reviews that appear in Wine Enthusiast magazine, admits that some types of wine qualify for his top two tiers of quality, rating 90 points or more, while others don’t. “Rose wines aim for early drinkability and do not aspire to greatness,” he says, “and White Zinfandel is not shooting for complexity either. We apply our quality scale universally, to all wine types.” By its nature, the best White Zin in the world would be unlikely to score 100.

But professor Meredith argues that nearness to perfection is not the only criterion for actually buying a wine. “You have got to factor in value when judging a wine. A low-priced wine might be simply clean and fruity, but those characteristics would represent high quality for that type of wine.”

Because quality ratings of wine are so subjective, the personal taste and experience of the people giving the rating is an important qualifier of every score. Individual critics such as Parker or Tanzer of International Wine Cellar newsletter have built followings of readers who are able to take the critics’ tastes into consideration. At Wine Spectator, the magazine’s specialist in each wine region is charged with writing the tasting notes for the wines of that area. “We try to objectify the tasting process by using experienced, articulate people as tasters,” Gordon explains. Beverage Testing Institute relies on a group of judges who have trained themselves to taste consistently “and who have a mental imprint of where the qualitative bars lie for every type of wine,” as Laverick puts it.

To counterbalance the actual imprecision of judging, most critics publish descriptions of each wine along with the wine’s score, and caution consumers not to take the quality rating out of context. “The numbers are not absolute, and cannot be followed blindly,” Gordon maintains. “We don’t claim to be wine gods; we just give our best advice and opinion on how good wines are.”

Ultimately, the business of judging wine is far too subjective for any wine buyer to swallow ratings or quality judgments wholesale, without considering his or her own tastes. Nor do critics expect that. “Readers have to trust their own judgment,” advises Gordon. “If someone tries a wine we recommend and doesn’t agree with our assessment, then he gets to know that critic’s views.”

Meredith concludes, “Consumers need to become familiar with the different sorts of wine evaluations out there and understand that not all of them are good predictors of their own wine preferences.”

* Ewing-Mulligan is a Master of Wine and wine educator. Her most recent book, written with Ed McCarthy, is “Wine Buying Companion for Dummies.”