The Nation : Science : Keeping Scientists Honest When Peer Review Fails

It was a victimless crime, and it didn’t make many headlines. Only someone who spends hours a day with her nose pressed against a computer screen would have noticed when, earlier this summer, a researcher formerly at the Lawrence Berkeley National Laboratory was found by federal watchdogs to have fabricated information on the health effects of electromagnetic fields. EMFs, as they’re known, are shrouds of ambient energy that surround computers, power lines and ordinary household appliances like auras. In the past decade, they have been accused of causing maladies ranging from miscarriage to depression.

In 1992, in two provocative and widely read reports, biochemist Robert P. Liburdy claimed to show how EMFs might trigger the rampant cell division that leads to cancer. According to the federal Office of Research Integrity, he manipulated some of the key data supporting his argument; some he outright made up.

The collective results of EMF studies are so inconclusive that revelations of Liburdy’s misconduct had almost no impact on the prevailing lack of wisdom. The tale of twisted data is chilling nonetheless, involving a health concern as gargantuan as cancer and a fixture of modern civilization as ubiquitous as electricity. The potential implications for public policy alone demand that any research brought to bear on the EMF debate be accurate and reliable or, at the very least, not make-believe. But for the conscience of a whistle-blower, Liburdy’s claims would have entered the annals of fact.

What does the public have to do to ensure the legitimacy of scientific evidence and the accountability of the scientific establishment?

Last fall, Congress passed a law that answers that question explicitly. Prompted by a tussle over clean-air statutes, it was more or less sneaked into an omnibus bill and met with cries of protest as soon as scientists got wind of it. In its native state, the law requires that federally funded investigators release their raw data to anyone who seeks them under the Freedom of Information Act, which guarantees public access to certain government documents. There’s no doubt that complying would be a pain in the Bunsen burner for most folks in the lab. But the standard system by which scientists police their own ranks is in need of some backup. A balanced interpretation of the law could restore credibility to research enterprises such as the EMF, clean-air and lead contingents, where it is desperately needed.

Scientists and lawmakers are currently engaged in a spirited effort to find that balance. Research advocates, including the National Institutes of Health, the National Academy of Sciences and the Assn. of American Universities, have complained that the legislation’s language is too sweeping. They fear it would infringe on scientists’ ability to work by interfering with data analysis, by compromising the privacy of individuals in clinical trials and by violating researchers’ intellectual-property rights or their proprietary arrangements with partner companies.

Beyond these reasonable objections, one can sense an almost aesthetic indignation. There’s a way in which demanding access to raw data--the computer printouts, the micrograph images, the careful entries in the notebook--is akin to scrutinizing a novelist’s first draft of a manuscript while it’s still being written. The data-access law also offends because scientists pride themselves on their ability to weed out sloppy, half-baked findings without anyone else’s help. The favored instrument of triage is the peer-review process, an august tradition in which a researcher’s work is vetted by colleagues before being deemed acceptable for publication. The recent legislation amounts to a no-confidence vote on peer review, and, as such, it threatens to erode science’s culture of self-sufficiency. But, as Liburdy’s case attests, peers can indeed be duped. The peer-review system may be too fallible to warrant public trust when scientific results are used to guide major policy decisions.

Interestingly, scientists may be coming around to that view. In a recent issue of the country’s leading science journal, analysts from UC Irvine and the American Enterprise Institute noted that several studies designed to gauge the efficacy of peer review have revealed an alarmingly imperfect process. In one study, a paper with eight deliberate errors was distributed to hundreds of reviewers. No one found more than five of the errors, and most found just two. “Even if the peer-review process were adequate for academic purposes, however, it is frequently not adequate for major public-policy decisions,” the authors concluded. For those decisions, they suggest that scientific evidence used to support major regulations be double-checked before the regulations are finalized.

In addition to the data-access measure, the analysts propose the creation of an agency that would replicate critical research, that is, repeat the experiments that yielded significant findings to make sure they give the same results a second time. This sort of exercise would have foiled Liburdy. The requirement for data release and replication would be limited to regulations that have an annual economic impact of at least $100 million, thereby sparing researchers the hassle unless the stakes were fairly high.

While one may quibble with the actual amount, the idea of naming a threshold below which verification would not be required makes sense. It’s another way in which scientists could be reassured that data-access law is a good-faith effort, not legislative harassment. Existing provisions in the Freedom of Information Act would also protect researchers from any gross abuses of the statute. Rules set forth this summer by the Office of Management and Budget, in response to scientists’ concerns, have already limited the scope of the mandate, making it more palatable to its opponents.

The comment period on these proposed rules will end Sept. 30; a revised law is likely to go into effect sometime after that. Whatever the practical impact, the image of the scientist as secretive genius scribbling indecipherable runes must be consigned to the past. It’s just as well. While declining to admit to any malfeasance, Liburdy agreed in May to retract his results and forego federal funding for three years. But by 2003, he could be on the payroll again.*