THE CUTTING EDGE: COMPUTING / TECHNOLOGY / INNOVATION : Peer Review Gets a Long, Critical Look
One of the great sacred cows of science, the “peer review” system that determines who gets published in professional journals and who gets funding for their work, is now facing a serious attack.
The system, in which proposals for publication and funding must first be approved by others in the same field, has always been unpopular with some who feel their work is not getting the support it deserves. The identity of the judges is secret, and some believe those who offer the harshest judgment are in fact their chief competitors.
My e-mail basket bulges with messages from scientists who feel they have been left out of a “good old boy” network and cannot get either published or funded. These complaints have always been around, but disaffection with the system has gone far beyond that in recent months.
The federal government’s General Accounting Office cited similar complaints in a report last year that challenged funding agencies to revamp their systems to “ensure fairness” in these times of tight dollars. The GAO stopped far short of recommending scrapping peer review, however, concluding that while it may be flawed, it is the best system we have.
Not everybody agrees with that.
“This thing is a real mess,” says Rustum Roy, founding director of the Materials Research Laboratory at Pennsylvania State University, and the most outspoken critic of peer review. “It doesn’t work.”
Roy says the current system is wasteful, ineffective, and does nothing to reduce the chances of scientific fraud. And he’s carrying out his debate in the public arena, going so far as to establish his own journal for materials scientists that will not submit papers to peer review. The first issue, called the MRS Bulletin, is now in the mail.
Others have taken to the courts to challenge the secrecy behind the peer review process. Wanda Henke, co-owner of a seismic technology firm in Maryland, sued the National Science Foundation and the National Institute of Standards and Technology, claiming that freedom of information laws require the release of the names of those who have judged her grant proposals. She won her case against NIST, but lost against the NSF; her attorney says they will appeal all the way to the Supreme Court, if necessary.
Roy is a veteran of the peer review process. He says he has submitted more than 3,000 proposals in what he now calls an “arcane ritual.”
He argues that a third of the nation’s scientific resources are consumed in the sometimes lengthy process of winning approval for funding or publication.
He says he has conducted an experiment at Pennsylvania State that proves his point. Twenty of his students were assigned to write proposals for funding, and the average time it took to research and write each proposal was a little more than one month. That adds up to nearly two years of time, he says, at a cost in overhead of about $250,000.
With the current approval rate of between 5% and 10%, they could expect one of the proposals to be funded, Roy says, for an average of $80,000.
“So we lost money,” he says.
Aside from the waste, Roy says the most serious failure of peer review is it does nothing to reduce fraud. “The entire history of fraudulent science has been under the peer review system,” he says.
“There is no accountability,” he adds. He claims no one ever checks to see if the funded research was ever carried out.
“If they did,” he says, “they would find that 50% of the people never did what they said they were going to do [with the grant].”
He says a better system would place accountability squarely on the shoulders of the applicant. Roy advocates a system based on “performance, not promise,” and he is applying that to his own journal.
He says funding and publication should depend on the track record of the applicant, not peer review of a single proposal.
Under that system, “if you make a mistake, you pay the price,” he says. Young scientists could enter the system by being sponsored by a seasoned veteran who knew them well enough to put his or her own reputation on the line. Roy says his system would eliminate the “huge bureaucracies” in Washington.
*
Recognizing that there are flaws in the system, the NSF is midway through a year-long study of peer review, but it is not even considering abandoning it. NSF Director Neal F. Lane recently called it the “best quality-control mechanism ever developed for investments in research and education.”
The purpose of the study is to “fine tune” the system, not overhaul it, according to an NSF official. The agency has an e-mail address for comments from scientists, effective until Dec. 1. The address is proprev@nsf.gov. Peer review is so entrenched that even Roy has had to rely on it to get his journal started. To reach a “qualification benchmark” for publication, he says, a scientist must have published at least 50 peer reviewed papers. However, the paper itself will not be judged by peers.
Even if he is on the right track, his difficulty in launching his own journal without drawing from a system he despises shows just how difficult it would be to change it. And couldn’t his system lead to an “old boy network” that is even more entrenched and more political than the one we have now?
Roy argues that would not be likely because each scientist would have to stand on the merits of his or her past performance, not the cleverness required to get a single proposal past peers.
Lee Dye can be reached via e-mail at 72040.3515@compuserve.com
