Scientists debate fixes -- and if they’re needed

Fiber wards off colon cancer, fiber doesn’t ward off colon cancer. Vitamin E protects the brain, vitamin E doesn’t protect the brain. If population studies have problems -- and many researchers think the flaws are being exaggerated -- what is the solution?

Scientists are scuffling about that, as well.

Stan Young, a statistician at the National Institute of Statistical Sciences in Research Triangle Park, N.C., says journals should ask their authors to perform careful statistical analysis that accounts for the habit of testing lots and lots of hypotheses, and also to share their data so others can check if the studies hold up.

When Young did a poll among journal editors, he found that few of them currently do.

Some journals are starting to address the situation. The Annals of Internal Medicine, for example, now has a section in the abstract that lists the limitations of a given study, says Dr. Cynthia Mulrow, deputy editor of the journal.

And in October, the Annals and several other journals will publish a statement calling for better reporting of the results of epidemiological studies. It will ask authors to voluntarily describe the questions and the protocol they used in the study, Mulrow says.

But these measures are voluntary, and some have been discussing more drastic measures until things improve.

Dr. Robert Temple, director at the office of drug evaluation at the Food and Drug Administration, says he wrote in an editorial a few years ago that epidemiological studies should only be published after they have been replicated at least once. (He’s not sure how this would be enforceable, though.)

Some scientists dislike this idea.

“I think that’s completely ridiculous,” says Dr. Meir Stampfer, a professor of epidemiology and nutrition at the Harvard School of Public Health and a professor of medicine at Harvard Medical School.

Observational studies should be published, Mulrow says -- they give important insights that scientists can then test more rigorously.

Others think these suggestions don’t go far enough. “You need to stop people from publishing epidemiological papers in the popular press,” says Nassim Nicholas Taleb, a former Wall Street trader and statistician who has written books about the role of chance.

His days on Wall Street, he says, have taught him that the more data there are the less likely it is to find anything significant. He says that if nothing is done to improve the situation, things may soon become even worse than Greek epidemiologist John Ioannidis’ estimate that 80% of epidemiological studies are false.

As the data pile up, he says, “This could soon be 99%.”

He, for his part, has already taken action.

“I haven’t read the paper since 1987,” he says.