“The first principle is that you must not fool yourself — and you are the easiest person to fool,” physicist Richard Feynman famously told the young scientists graduating from CalTech in 1974. Fully cognizant of this truth, the scientific establishment has developed many rules and procedures to weed out false findings from experiments, key among them replication.
Replication means that an experiment can be repeated over and over, by the original researcher or any other competent scientist in the field, and it will produce the same or similar result. Now, science is in the midst of a “replication failure” crisis — at least according to scores of articles in the scientific and mainstream media.
Is science truly in trouble? Rife with fraud? Losing reliability?
Absolutely not. Science is doing what it always has done — failing at a reasonable rate and being corrected. Replication should never be 100%. Science works beyond the edge of what is known, using new, complex and untested techniques. It should surprise no one that things occasionally come out wrong, even though everything looks correct at first.
The mistake is to think that any published paper or journal article is the end of the story and a statement of incontrovertible truth. It is a progress report.
Replication failures should not be conflated with scientific fraud, which is rightly condemnable. The failure to replicate a part or even the whole of an experiment is not sufficient for indictment of the initial inquiry or its researchers. Failure is part of science. Without failures there would be no great discoveries.
How then should we respond to replication failures? They should be published without prejudice. In science, revision is a victory — not a devious cover-up or intellectual flip-flop. Yes, a complete inability to reproduce results could indicate an overlooked fatal flaw in the study. But it more often stems from subtle inconsistencies between one experiment and the next. Pinpointing that inconsistency is how we discover what we didn’t even know that we didn’t know.
For example, in the early 20th century controversy raged over how nerves made muscles and glands respond. Was it bio-electricity or chemicals? In 1921 an Austrian biologist, Otto Loewi, dreamed, literally, of a simple experiment that would settle the issue and took to his lab in the middle of the night to test it.
Then, Loewi let the solution surrounding first heart flow into the second heart’s liquid bath. Shortly, the second, nerveless heart also began to slow. Loewi’s concluded that the stimulated vagus nerve released a chemical that caused the first heart muscle to slow its contractions — and then that chemical seeped into the saline and had the same effect on the second heart. In short, Loewi had proven that neurotransmission was inherently chemical, not electrical.
Except — this simple and brilliant experiment couldn’t be replicated, even by Loewi, for nearly six years. Why? Loewi had done his first experiment in the cold night, and the other replications were all done during warmer days or in heated buildings. And that mattered. First, frogs’ physiology changes seasonally: their heart rate is less susceptible to modulation in the spring and summer. Second, the chemical transmitter (now known to be acetylcholine) gets broken down by an enzyme that is more active when it is warm.
What science learned from this replication failure was that physiology can be seasonal and that enzymes are modulated by temperature — and eventually how synapses fire. In 1936 Loewi shared a Nobel prize for this discovery.
Replication failure is more common in newer areas of science than in the mature fields. It is now less common in astronomy, physics and many branches of chemistry, while it seems to plague organismic or systems biology, psychology and social psychology in particular. The younger the field the less we know about the variables that can fool us when we don’t control for them.
Science would be in a crisis if it weren’t failing most of the time. Science is full of wrong turns, unconsidered outcomes, omissions and, of course, occasional facts. Replication is part of that process, as open to failure as any other step. The mistake is to think that any published paper or journal article is the end of the story and a statement of incontrovertible truth. It is a progress report. Don’t be fooled.
Stuart Firestein is the former chair of the Department of Biological Sciences at Columbia University and the author of “Failure: Why Science is so Successful.”