Polls Should Be Messengers, Not Messages

Public opinion polls are one of the boogeymen of the 1996 presidential election. The indictment against polls is that they are fickle, unreliable and a distraction from thinking about politics. The Wall Street Journal lashed out the day after the election that “the disparities in the final polls were dramatic, ranging from a Clinton lead of 18 points in the New York Times/CBS poll down to a Clinton lead of only 7 points in the Reuters/Zogby poll.” Fussing over how much Clinton was up or down, the Journal charged, “substituted for thinking about politics.’

The skepticism toward horse race polling is understandable, but the blame has been misplaced. The problem lies with the media. Journalists--in their scramble to package a drama-filled story of a changing race--oversold, misrepresented and misused polls.

The most important thing to know about polls is that they are estimates. They are not the precision instruments of nuclear engineering. Modern polling rests on the drawing of a random sample of the population; demographically weighted random samples as small as 500 or 1,000 adults are used to estimate the views and behaviors of the entire adult population.

During the 1996 election, no one knew President Clinton’s true lead over Bob Dole and Ross Perot; that would have required interviewing all actual voters. As a substitute, the media, the candidates and the American public all turned to pollsters for estimates of the candidates’ standings. It is natural and indeed expected that these estimates of Clinton’s lead would be distorted by predictable and quantifiable errors in the survey process.

Because random samples do not exactly mirror the overall population, a single survey of 600 to 750 “likely” voters will produce estimates of a candidate’s “true” level of support that can vary by plus or minus 4 percentage points. A national poll of 600 to 750 voters that showed 50% of respondents choosing Clinton and 40% Dole should be interpreted as offering an estimate within a full 8-point range: The gap could be 2 points (46% to 44%) or 18 points (54% to 36%).

Most surveys conducted just before election day were within sampling error of the actual results. Unlike many other polling organizations, the New York Times/CBS stopped its survey on Saturday and missed the late shift of support from Clinton to Dole and Perot during the last three days of the campaign.

The most striking feature of horse race polls in 1996 was the remarkable stability of Clinton’s lead in hundreds of surveys. Discrepancies that showed a narrow Clinton lead and that landed in the headlines were by far the exceptions.

Our analysis of a sample of 129 newspaper and broadcast stories during the fall presidential campaign found that journalists’ fixation with horse race polls soaked up time from exploring ample polling results on policy issues and other matters. More than seven out of 10 stories mentioned polls that sized up Clinton’s lead.

In the news media’s effort to present the election as tightening and capture their audience’s attention, they persistently distorted polls. For instance on Oct. 12, CNN anchor Joie Chen declared that “the race may be getting tighter” based on a Gallup tracking poll that found a 3-point drop in Clinton’s standing (54% to 51%) and a 2-point rise in Dole’s support (36% to 38%). In fact, that shift was statistically insignificant because it was easily accounted for by sampling error.

Journalists largely report only their own polls except when other organizations poll on topics--such as state races--for which they have no internal results. Seven out of 10 television stories that discussed poll results related to the presidential campaign used only internal results. CNN and ABC effectively treat their polls as the first and last word on public opinion. In effect, each news outlet is declaring that its poll represents a monopoly of truth.

The insulated approach to polling is driven by a kind of protectionism: Favor one’s own polls and resist giving added exposure to the results of competitors. The losers are audiences, who are denied the necessary basis for comparing results and reaching more accurate conclusions.

The flood of polls is not a disservice to the public; rather, it is how the polls have been used that is the problem. Indeed, the proliferation of polling offers an opportunity for challenging questionable polls or media interpretations. Possible solutions: regular poll reports by journalists or the increasingly widespread websites that use the wealth of polls to identify flawed polls or distorted media coverage of polls. One job for these poll watches is to review such technical issues as wording or the drawing of samples. Another is to openly compare new results with polls from other sources. Finally, poll watches would challenge the media fixation with reporting horse race polls and draw attention to the extensive (but infrequently reported) polls on policy issues and other matters.