BUSINESS PULSE / JOHN BRENNAN : How It Is That ‘Scientific’ Surveys’ Results Can Be Polls Apart

George H. Gallup, an inventor of modern opinion polling, idealistically believed that surveys of the public would enhance democracy by providing leaders with a true picture of peoples’ attitudes and concerns.

Sixty years have gone by, and polls are now as common as Big Macs on the American landscape. But their effect remains dubious at best, as does the effect of the burgeoning number of studies of all types that are now part of every public policy debate.

In theory, research using the scientific method tries to measure a phenomenon (such as public opinion, health risk or environmental impact) in the most objective manner possible. Of course, nothing is ever completely unbiased. But good researchers strive for that goal.

There’s ample room for their efforts. Public policy debates abound with myths and ideology, often based on nothing but hearsay. Good research can cut through these shibboleths and keep the press and policy makers from making wrong assumptions. Lately, for example, careful polling has shown the lack of evidence for the existence of an angry, alienated Generation X, that mythical group of twentysomethings so beloved by scores of feature writers.

Unfortunately, an increasing amount of research is being conducted to support particular myths rather than to seek objective answers. In her new book “Tainted Truth: The Manipulation of Fact in America,” Wall Street Journal writer Cynthia Crossen chronicles the ominous rise of this “tactical” or “advocacy” research, in which groups wanting to advance their agendas commission supposedly objective studies that almost always end up supporting their positions on an issue.

Crossen cites numerous examples, including a research battle between advocates of cloth and disposable diapers, in which each side weighed in with impressive environmental impact studies showing diametrically opposite results.

According to Crossen, advocacy research is “contributing to the steady degradation of social discourse on the major political, economic and environmental issues of the day.” Anyone listening to media debates on health care, illegal immigration, abortion or the North American Free Trade Agreement could agree. Each side seems to have its own ready-made version of statistical reality, one the total opposite of the other.

When tactical research is identified as sponsored, the observer can at least consider the source and tread warily. More insidious, perhaps, are the occasions in which advocates manipulate independent findings or cite only selected research results to advance a cause.

Such a charge was recently leveled at the U.S. Environmental Protection Agency, which has been accused of using some questionable techniques to determine that secondhand cigarette smoke is a Class A carcinogen. The EPA’s actions drew a not-surprising attack from the tobacco industry. But they also earned criticism from the Congressional Research Service, which has raised doubts about some of the methodology. Nonetheless, this research remains the basis for the government’s goal to ban smoking in virtually all workplaces.

A different but equally worrisome trend is the growing use of unscientific “gimmick” studies, primarily for promotion, publicity or “just for fun.” Evidence of this is the recent revival by CBS and NBC news programs of the long-discredited 900-number call-in technique. Such call-ins produce biased, unrepresentative findings, and can be manipulated by interest groups manning phone banks. Both networks maintain top-notch polling units that can deliver sound scientific surveys in hours. But these lack the dramatic allure of letting the audience vote during the program. The result is the airing of scurrilous data that misleads rather than informs.

In the same sphere are the new so-called on-line “polls,” in which subscribers to on-line computer systems log in their views to a question and the system reports the results. These are particularly skewed: Computer on-line systems are currently used by only a fraction of the country, yet findings often are reported as if they represent America as a whole. The public--and much of the press--has a hard time discerning that fact.

One might expect journalists to sound the alarm about the pitfalls of bad research. Many--such as Crossen--are doing just that. But the industry as a whole does a woefully inadequate job. Impressive-looking studies are treated as breaking news. And competitive pressures often push news people to “go with it now and ask questions later.”

Although much good research is being done, we are a long way from George Gallup’s dream of survey-assisted democracy. On the contrary, it is the average citizen who must be on guard to judge cautiously the dizzying array of statistics coming his or her way.