Study: Psychiatric Research Fails ‘Real World’ Test
Psychiatric research often weeds out the very people who would benefit most from it, a new study suggests, leading to the development of treatments that often fail in the real world.
The studies also disproportionately exclude blacks, poor people and those with severe drug and alcohol problems, the researchers report in this month’s American Journal of Psychiatry.
“It’s a problem in medicine as a whole and is by no means limited to psychiatry,” said Dr. Paul Appelbaum, vice president of the American Psychiatric Assn.
The researchers--Keith Humphreys of Stanford University Medical Center and Constance Weisner of UC San Francisco--said their findings may explain why treatments that perform well in controlled studies often are less effective outside the lab.
“The kinds of people who end up in research studies aren’t the ones who are seeking care,” said Humphreys, who sees no evil intent behind exclusion criteria.
“I don’t believe for a minute that this is done on purpose,” he said. Researchers are motivated by legitimate concerns, such as convenience and cost, as well as the desire for a smooth-running, successful study, he said.
The two professors in psychiatry wanted hard evidence of this problem, which researchers throughout medicine have acknowledged. So they went to 593 people in eight alcohol treatment clinics in Northern California and applied eight exclusion criteria commonly used in alcohol treatment clinical trials to see how many of the real patients would qualify for a hypothetical study.
The criteria they considered rule out people who have psychiatric or emotional problems, have serious medical problems, live too far from the treatment center, are dependent on nonprescribed drugs, are single and unemployed, are homeless, have failed to respond to prior alcohol treatment, or for lack of motivation or some other reason are thought not likely to comply with the demands of a controlled study.
Depending on which criterion was applied, the researchers found that between 8% and 75% of the patients would be disqualified. The criterion disqualifying the fewest people was being noncompliant or unmotivated. The one that ruled out the most was failure at previous treatment.
For each criterion, the researchers calculated the proportion of patients it would exclude, then compared the characteristics of the excluded and included patients. Researchers found significant difference between the groups at both public and private clinics.
Blacks at both clinics were disproportionately excluded when the drug dependence criterion was applied.
People from both clinics who earned less than $10,000 were more likely to be excluded from studies because of several of the criteria. The same was true for patients who were dependent on drugs or had severe alcohol or psychiatric problems.
Gender had little impact.
There are times when exclusion criteria are necessary, Humphreys and Weisner concluded, but research can’t be considered complete until studies with few or no exclusion criteria are conducted.
Applying exclusion criteria is legitimate in initial studies because their purpose is to determine if a specific treatment works, said Appelbaum, chairman of the psychiatry department at the University of Massachusetts Medical School. To do that, factors that may affect the outcome need to be limited so the treatment’s effect can be isolated.
“The problem comes when that initial approach is not followed up with studies that are more real-world oriented,” Appelbaum said. And often, they are not.
One reason is that pharmaceutical companies sponsor most drug studies and they don’t have the incentive to do these kinds of follow-up studies, Appelbaum said.
The Food and Drug Administration, which approves new medication, could help correct the problem by requiring drug companies to demonstrate some degree of “real world” effectiveness, Appelbaum said.
Also, he said, the federal government should consider funding this type of research, which, Humphreys noted, would be costly.
