Even for Veteran Research Firm, It Was New Territory

Shortly after Al Gore conceded last year’s cliffhanger presidential contest to George W. Bush 36 days after election day, eight media companies began a joint effort to determine why so many Florida voters had cast spoiled ballots.

Putting aside traditional rivalries, six national news organizations and two Florida newspapers hired the National Opinion Research Center, or NORC, a nonpartisan institute at the University of Chicago, to record every mark on every ballot rejected by tabulating machines or county canvassing boards. The ballots were available under Florida’s public records law.

Joining forces were the Tribune Co., which owns the Los Angeles Times, Chicago Tribune, Newsday, Orlando Sentinel and South Florida Sun-Sentinel; the New York Times; the Washington Post; Associated Press; Cable News Network; the Wall Street Journal; the St. Petersburg Times; and the Palm Beach Post. Not including reporting and some legal bills, the project cost about $900,000.

NORC’s Chicago-based researchers had never studied disputed presidential ballots. No one had.

But NORC had done more than 1,000 research surveys of health, education, drug abuse, crime and other issues for government agencies and private foundations. The group assigned 20 staffers to the effort. They, in turn, hired, trained and supervised 153 others to study the ballots.

The goal was to examine all the undervotes--ballots that had registered no vote for president--and all the overvotes--ballots disqualified because they showed more than one choice. Beginning Feb. 5, the NORC teams visited all 67 Florida counties, some several times, but county officials could not recover about 1,400 uncounted ballots. They thus are not part of the study.

The researchers, called “coders,” were not allowed to handle ballots. County officials held them up for study or put them on light boxes for closer inspection. And from the start, the researchers were instructed to record only ballot marks--not to count votes.

“We banished the word ‘counting’ from the vocabulary,” said Tracy Buie, a NORC supervisor. “We were collecting data.”

Newspapers have published more than a dozen ballot studies since last fall. But no two examined the same ballots or used the same analytical standards.

The only other statewide ballot review was conducted by USA Today, the Miami Herald and six other Florida newspapers. Like the current study, they tried to examine all undervotes and overvotes. But their methodology differed in key ways.

Those newspapers used an accountant, aided by a reporter, to inspect undervotes. Reporters alone examined the overvotes. In some cases, they relied on county computer records rather than visually inspecting the ballots.

The NORC teams examined about 5,000 more ballots than the Herald/USA Today study. NORC also assigned three people to inspect each undervote simultaneously. That way, their data could be cross-checked to assess its reliability, said Kirk Wolter, senior vice president at NORC and a professor of statistics at the University of Chicago.

On the overvotes, however, the news groups agreed to let NORC assign only a single researcher--not three--after a field test in several counties. The test showed obvious multiple marks on virtually every overvote.

For undervotes, the news groups regarded a 2-1 vote as evidence of a valid ballot. That is, it required at least two researchers to acknowledge the same mark or better for a ballot to be considered a vote in scenarios that accepted that mark as evidence of voter intent.

In only one of the possible recount scenarios would the result have been different if unanimous agreement by the three researchers had been required. In the scenario that counts every dimple on punch cards and every affirmative mark on optical scan ballots, Gore wins when agreement by two researchers makes a ballot valid, but Bush wins when the standard is unanimity and fewer ballots pass muster.

Safeguards built into the project helped identify two potential problems. A Baker County researcher noted marks for Bush on several dozen undervotes that appeared empty to her two teammates. Her work was not suspect at the time, but she was fired on her first day for failing to follow directions.

NORC analyzed its own data over the summer to see how often coders agreed with one another. Inconsistencies would help identify any “rogue” coders or spoiled data. The Baker County coder stood out because she disagreed with her colleagues most of the time.

Because the two other researchers had agreed in nearly every case, the woman’s data did not affect vote tallies.

In the second case, a former NORC coder during the summer posted an anti-Bush commentary on a Web site for Democratic Party activists. Although he had reviewed more than 4,000 ballots in six counties, a NORC analysis of his ballot data showed he had agreed with his table mates in almost every case. So his politics had no discernible effect on the study.

NORC checked whether researchers behaved differently depending on their personal characteristics, including age, gender and political affiliation. Wolter, director of the project, said men tended to count more undervotes as valid--for either Bush or Gore--than did women. He said researchers who identified themselves as Republicans reported seeing more undervotes for Bush than Democrats did--even though the candidates’ names were not on the punch cards they saw.

But he said he did not believe this was a result of bias. “At the margin, when you’re talking about whether you saw a slight dimple or not, there is room for some judgment,” he said. “People will . . . honestly see it in different ways.”

In any case, he noted, any variation in how researchers evaluated dimples would not affect recount standards that do not count dimples as valid votes. The study’s overall finding--that Bush probably would have won a recount under last year’s rules, but Gore might win under this year’s rules--holds true even if dimples are ignored.

Of 153 researchers who examined the ballots, 36% said they were Democrats, 30% said they were Republicans and 30% said they were independents--a division similar to that of Florida’s registered voters.

Republican Party operatives shadowed the research teams at the start and criticized the study as inaccurate, invalid and inappropriate. “The one standard the media has used is no standard whatsoever,” charged Stephanie Housel, a GOP official.

The news groups initially hoped to finish by April. But legal challenges, logistical problems in recovering ballots and other delays meant field work didn’t end until May 30. Preparation of computer programs needed to analyze the data took much of the summer. Plans to publish in mid-September were postponed because so many reporters and editors were covering the Sept. 11 terrorist attacks and their aftermath.

(BEGIN TEXT OF INFOBOX / INFOGRAPHIC)

(text of infobox not included)