FBI resists scrutiny of ‘matches’
ABOUT THIS SERIES
This is the second in a series of occasional articles that will examine how DNA evidence is transforming criminal justice.
State crime lab analyst Kathryn Troyer was running tests on Arizona’s DNA database when she stumbled across two felons with remarkably similar genetic profiles.
The men matched at nine of the 13 locations on chromosomes, or loci, commonly used to distinguish people.
The FBI estimated the odds of unrelated people sharing those genetic markers to be as remote as 1 in 113 billion. But the mug shots of the two felons suggested that they were not related: One was black, the other white.
In the years after her 2001 discovery, Troyer found dozens of similar matches -- each seeming to defy impossible odds.
As word spread, these findings by a little-known lab worker raised questions about the accuracy of the FBI’s DNA statistics and ignited a legal fight over whether the nation’s genetic databases ought to be opened to wider scrutiny.
The FBI laboratory, which administers the national DNA database system, tried to stop distribution of Troyer’s results and began an aggressive behind-the-scenes campaign to block similar searches elsewhere, even those ordered by courts, a Times investigation found.
At stake is the credibility of the compelling odds often cited in DNA cases, which can suggest an all but certain link between a suspect and a crime scene.
When DNA from such clues as blood or skin cells matches a suspect’s genetic profile, it can seal his fate with a jury, even in the absence of other evidence. As questions arise about the reliability of ballistic, bite-mark and even fingerprint analysis, genetic evidence has emerged as the forensic gold standard, often portrayed in courtrooms as unassailable.
But DNA “matches” are not always what they appear to be. Although a person’s genetic makeup is unique, his genetic profile -- just a tiny sliver of the full genome -- may not be. Siblings often share genetic markers at several locations, and even unrelated people can share some by coincidence.
No one knows precisely how rare DNA profiles are. The odds presented in court are the FBI’s best estimates.
The Arizona search was, in effect, the first test of those estimates in a large state database, and the results were surprising, even to some experts.
Lawyers seek searches
Defense attorneys seized on the Arizona discoveries as evidence that genetic profiles match more often than the official statistics imply -- and are far from unique, as the FBI has sometimes suggested.
Now, lawyers around the country are asking for searches of their own state databases.
Several scientists and legal experts want to test the accuracy of official statistics using the 6 million profiles in CODIS, the national system that includes most state and local databases.
“DNA is terrific and nobody doubts it, but because it is so powerful, any chinks in its armor ought to be made as salient and clear as possible so jurors will not be overwhelmed by the seeming certainty of it,” said David Faigman, a professor at UC Hastings College of the Law, who specializes in scientific evidence.
FBI officials argue that, under their interpretation of federal law, use of CODIS is limited to criminal justice agencies. In their view, defense attorneys are allowed access to information about their specific cases, not the databases in general.
Bureau officials say critics have exaggerated or misunderstood the implications of Troyer’s discoveries.
Indeed, experts generally agree that most -- but not all -- of the Arizona matches were to be expected statistically because of the unusual way Troyer searched for them.
In a typical criminal case, investigators look for matches to a specific profile. But the Arizona search looked for any matches among all the thousands of profiles in the database, greatly increasing the odds of finding them.
As a result, Thomas Callaghan, head of the FBI’s CODIS unit, has dismissed Troyer’s findings as “misleading” and “meaningless.”
He urged authorities in several states to object to Arizona-style searches, advising them to tell courts that the probes could violate the privacy of convicted offenders, tie up crucial databases and even lead the FBI to expel offending states from CODIS -- a penalty that could cripple states’ ability to solve crimes.
In one case, Callaghan advised state officials to raise the risk of expulsion with a judge but told the officials that expulsion was unlikely to actually happen, according to a record of the conversation filed in court.
In an interview with The Times, Callaghan denied any effort to mislead the court.
The FBI’s arguments have persuaded courts in California and other states to block the searches. But in at least two states, judges overruled the objections.
The resulting searches found nearly 1,000 more pairs that matched at nine or more loci.
“I can appreciate why the FBI is worried about this,” said David Kaye, an expert on science and the law at Arizona State University and former member of a national committee that studied forensic DNA.
But “people’s lives do ride on this evidence,” he said. “It has got to be explained.”
Concerned about errors
From her first discovery in 2001, Troyer and her colleagues in the Arizona Department of Public Safety’s Phoenix DNA lab were intrigued.
At the time, many states looked at only nine or fewer loci when searching for suspects. (States now commonly attempt to compare 13 loci, and they may be able to search for more in the future. But even now, in many cases, fewer than 13 loci are discernible from crime scene evidence because of contamination or because of degradation over time.)
Based on Troyer’s results, she and her colleagues believed that a nine-locus match could point investigators to the wrong person.
“We felt it was interesting and just wanted people to understand it could happen,” said Troyer, who initially declined to be interviewed, then cautiously discussed her findings by telephone, with her bosses on the line.
“If you’re going to search at nine loci, you need to be aware of what it means,” said Todd Griffith, director of the Phoenix lab. “It’s not necessarily absolutely the guy.”
Troyer made a simple poster for a national conference of DNA analysts. It showed photos of the white man and the younger black man next to their remarkably similar genetic profiles.
Some who saw the poster said they had seen similar matches in their own labs.
Bruce Budowle, an FBI scientist who specializes in forensic DNA, told colleagues of Troyer that such coincidental matches were to be expected.
Three years later, Bicka Barlow, a San Francisco defense attorney, came across a description of Troyer’s poster on the Internet.
Its implications became clear as she prepared to defend a client accused of a 20-year-old rape and murder.
A database search had found a nine-locus match between his DNA profile and semen found in the victim’s body. Based on FBI estimates, the prosecutor said the odds of a coincidental match were as remote as 1 in 108 trillion.
Recalling the Arizona discovery, Barlow wondered if there might be similar coincidental matches in California’s database -- the world’s third-largest, with 360,000 DNA profiles at the time. The attorney called Troyer in Phoenix to learn more.
Troyer seemed eager to talk about her discovery, which still had her puzzled, Barlow recalled. The analyst told Barlow she had searched the growing Arizona database since the conference and found more pairs of profiles matching at nine and even 10 loci.
Encouraged, Barlow subpoenaed a new search of the Arizona database. Among about 65,000 felons, there were 122 pairs that matched at nine of 13 loci. Twenty pairs matched at 10 loci. One matched at 11 and one at 12, though both later proved to belong to relatives.
Barlow was stunned. At the time, such matches were almost unheard of.
That same year, Fred Bieber, a Harvard professor and expert in forensic DNA, testified in an unrelated criminal case that just once had he seen a pair of profiles matching at nine of 13 markers, and they belonged to brothers. He had heard of a 10-locus match between two men, but it was the result of incest -- a man whose father was also his older brother.
Indeed, since 2000, the FBI has treated certain rare DNA profiles as essentially unique -- attributable to a single individual “to a reasonable degree of scientific certainty.”
Other crime labs have adopted the policy, and some no longer tell jurors there is even a possibility of a coincidental match.
Soon after Barlow received the results, Callaghan, the head of the FBI’s DNA database unit, reprimanded Troyer’s lab in Phoenix, saying it should have sought the permission of the FBI before complying with the court’s order in the San Francisco case.
Asked later whether Callaghan had threatened her lab, Troyer said in court, “I wouldn’t say it’s been threatened, but we have been reminded.”
Dwight Adams, director of the FBI lab at the time, faxed Griffith, Troyer’s boss, a letter saying the Arizona state lab was “under review” for releasing the search results.
“While we understand that the Arizona Department of Public Safety, acting in good faith, complied with a proper judicial court order in the release of the nine-loci search of your offender DNA records, this release of DNA data was not authorized,” Adams wrote, asking Arizona to take “appropriate corrective action.”
Arizona officials obtained a court order to prevent Barlow from sharing the results with anyone else.
But it was too late. After a judge found the Arizona results to be irrelevant in Barlow’s case, the defense attorney e-mailed them to a network of her colleagues and DNA experts around the country.
Soon, defense lawyers in other states were seeking what came to be known as “Arizona searches.”
For years, DNA’s strength in the courtroom has been the brute power of its numbers. It’s hard to argue with odds like 1 in 100 billion.
Troyer’s discovery threatened to turn the tables on prosecutors. At first blush, the Arizona matches appeared to contradict those statistics and the popular notion that DNA profiles, like DNA, were essentially unique.
Law enforcement experts scrambled to explain.
Three months after the court-ordered search in Arizona, Steven Myers, a senior DNA analyst at the California Department of Justice, gave a presentation to the Assn. of California Crime Lab Analysts. It was titled “Don’t Panic” -- a hint at the alarm Troyer’s discovery had set off.
Many of the Arizona matches were predictable, Myers said, given the type of search Troyer had conducted.
In a database search for a criminal case, a crime scene sample would have been compared to every profile in the database -- about 65,000 comparisons. But Troyer compared all 65,000 profiles in Arizona’s database to each other, resulting in about 2 billion comparisons. Each comparison made it more likely she would find a match.
When this “database effect” was considered, about 100 of the 144 matches Troyer had found were to be expected statistically, Myers found.
Troyer’s search also looked for matches at any of 13 genetic locations, while in a real criminal case the analyst would look for a particular profile -- making a match far less likely.
Further, any nonmatching markers would immediately rule out a suspect. In the case of the black and white men who matched at nine loci, the four loci that differed -- if available from crime scene evidence -- would have ensured that the wrong man was not implicated.
The presence of relatives in the database could also account for some of Troyer’s findings, the FBI and other experts say. Whether that’s the case would require cumbersome research because the databases don’t contain identifying information, they say.
Flaws in assumptions?
Some scientists are not satisfied by these explanations. They wonder whether Troyer’s findings signal flaws in the complex assumptions that underlie the FBI’s rarity estimates.
In the 1990s, FBI scientists estimated the rarity of each genetic marker by extrapolating from sample populations of a few hundred people from various ethnic or racial groups. The estimates for each marker are multiplied across all 13 loci to come up with a rarity estimate for the entire profile.
These estimates make assumptions about how populations mate and whether genetic markers are independent of each other. They also don’t account for relatives.
Bruce Weir, a statistician at the University of Washington who has studied the issue, said these assumptions should be tested empirically in the national database system.
“Instead of saying we predict there will be a match, let’s open it up and look,” Weir said.
Some experts predict that given the rapid growth of CODIS, such a search would produce one or more examples of unrelated people who are identical at all 13 loci.
Such a discovery was once unimaginable.
In January 2006, not long after Barlow distributed the results of the court-ordered search in Arizona, the FBI sent out a nationwide alert to crime labs warning of similar defense requests.
Soon after, the bureau’s arguments against the searches were being made in courtrooms around the country.
In California, Michael Chamberlain, a state Department of Justice official, persuaded judges that such a search could have “dire consequences” -- violating the privacy of convicted offenders, shutting down the database for days and risking the state’s expulsion from the FBI’s national DNA system. All this for a search whose results would be irrelevant and misleading to jurors, Chamberlain argued.
When similar arguments were made in an Arizona case, the judge ruled that the search would be “nothing more than an interesting deep sea fishing expedition.”
But in Illinois and Maryland, courts ordered the searches to proceed, despite opposition from the FBI and state officials at every turn.
In July 2006, after Chicago-area defense attorneys sought a database search on behalf of a murder suspect, the FBI’s Callaghan held a telephone conference with Illinois crime lab officials.
The topic was “how to fight this,” according to lab officials’ summary of the conversation, which later became part of the court record.
Callaghan suggested they tell the judge that Illinois could be disconnected from the national database system, the summary shows. Callaghan then told the lab officials “it would in fact be unlikely that IL would be disconnected,” according to the summary.
In an interview, Callaghan disputed he said that.
“I didn’t say it was unlikely to happen,” he said. “I was asked specifically, what’s the likelihood here? I said, I don’t know, but it takes a lot for a state to be cut off from the national database.”
A week later, the judge ordered the search. Lawyers for the lab then took the matter to the Illinois Supreme Court, arguing in part that Illinois could lose its access to the federal DNA database. The high court refused to block the search.
The result: 903 pairs of profiles matching at nine or more loci in a database of about 220,000.
State officials obtained a court order to prevent distribution of the results. The Times obtained them from a scientist who works closely with the FBI.
A ‘unilateral decision’
A similar fight occurred in a death penalty case in Maryland during the summer and fall of 2006.
The prosecutor saw a DNA match between a baseball cap dropped at the crime scene and the suspect as so definitive that he didn’t plan to tell the jury about the chance of a coincidental match, records show.
Seeking to cast doubt on the evidence, the defense persuaded the judge to order an “Arizona search” of the Maryland database. The state did not comply.
After the defense filed a contempt-of-court motion, Michelle Groves, the state’s DNA administrator, argued in court and in an affidavit that, based on conversations with Callaghan at the FBI, she believed the request was burdensome and possibly illegal.
According to Groves, Callaghan had told her that complying with the court order could lead Maryland to be disconnected from CODIS -- a result Groves’ lawyer said would be “catastrophic.”
Groves’ affidavit was edited by FBI officials and the technology contractor that designed CODIS, court records show. Before submitting the affidavit, Groves wrote the group an e-mail saying, “Let’s see if this will work,” the records show.
It didn’t. After the judge, Steven Platt, rejected her arguments, Groves returned to court, saying the search was too risky. FBI officials had now warned her that it could corrupt the entire state database, something they would not help fix, she told the court.
Platt reaffirmed his earlier order, decrying Callaghan’s “unilateral” decision to block the search.
“The court will not accept the notion that the extent of a person’s due process rights hinges solely on whether some employee of the FBI chooses to authorize the use of the [database] software,” Platt wrote.
The search went ahead in January 2007. The system did not go down, nor was Maryland expelled from the national database system.
In a database of fewer than 30,000 profiles, 32 pairs matched at nine or more loci. Three of those pairs were “perfect” matches, identical at 13 out of 13 loci.
Experts say they most likely are duplicates or belong to identical twins or brothers. It’s also possible that one of the matches is between unrelated people -- defying odds as remote as 1 in 1 quadrillion.
Maryland officials never did the research to find out.
(BEGIN TEXT OF INFOBOX)
As databases grow, so do the chances of finding a coincidental match. Three states have searched their DNA databases for pairs of profiles that have nine or more genetic markers in common. The more profiles in the database, the more matches were found.
Maryland: 33 matches in a database of 20,000 profiles
Arizona: 144 matches in a database of 65,000 profiles
Illinois: 903 matches in a database of 230,000 profiles
California: State database has more than 1 million profiles. Several search requests have been denied.
FBI: The national DNA database, maintained by the FBI, has almost 6 million DNA profiles. It has never been searched for such coincidental matches.
(BEGIN TEXT OF INFOBOX)
Experts use an analogy called the birthday paradox to explain that the way you search for a DNA profile can dramatically affect your chances of finding a match. In some circumstances, matches are far more likely than many people think.
Imagine you’re at a party with 99 other guests. If you randomly pull aside one of them, the odds he or she will share your day and month of birth are 1 in 365.
But the probability that anyone at the party shares your birthday is far higher: about 1 in 4. When you compare your birthday with 99 other people’s, each comparison makes a match more likely. (The math: Multiply the odds of 1/365 by 99, the number of comparisons, to get the approximate probability.)
For the same reason, the odds that anyone at the party shares a birthday with anyone else are higher still. In fact, it’s almost a certainty. As everyone looked for a match with everyone else, they made 4,950 comparisons. (The math: Multiply 100 people by the 99 other guests they compare themselves with, then divide by two because people who compare with each other count as a single comparison.)
How many people need to be at the party for it to be likely that two guests share a birthday?
The answer may surprise you: just 23.