A Nevada man sentenced to life in prison for sexually assaulting a young girl should be freed or given a new trial because a prosecution expert exaggerated the strength of the DNA evidence against him, the U.S. 9th Circuit Court of Appeals ruled Monday.

The ruling is thought to be one of the first of its kind in the country, experts said.

In 1994, Troy Don Brown, 36, was found guilty of sexually assaulting a 9-year-old girl in her family's trailer home in Carlin, Nev. The prosecution's case hinged almost entirely on semen samples collected at the scene that a DNA expert from the local sheriff's crime lab testified had a "99.99967% chance" of being Brown's, according to the appellate court's decision.

Brown was convicted. After unsuccessful appeals in state courts, he filed a petition in 2004 in federal court to be freed. Brown's lawyers argued that the state's DNA expert greatly inflated the mathematical probability that the DNA sample belonged to Brown.

After hearing evidence by a DNA expert who rebutted the prosecution's statistics, a lower court judge agreed with Brown. The 2-1 ruling Monday, written by Judge Kim Wardlaw and joined by Judge Michael Hawkins, upheld that decision.

Judge Diarmuid O'Scannlain, in a dissent, said the majority had failed to "give any weight to the DNA evidence," which he described as "overwhelming."

Nevada Atty. Gen. Catherine Cortez Masto could not be reached for comment. She must now decide whether to ask for another hearing before the 9th Circuit, appeal to the U.S. Supreme Court, retry Brown or release him from prison.

"It is rare for this argument to be made at all, largely because [lawyers] are often not sophisticated enough to understand it," said William C. Thompson, chairman of the Department of Criminology, Law and Society at UC Irvine.

Thompson and other experts in DNA evidence said it was the first case in the United States they knew of in which a conviction was overturned based on a mistake that in academic circles has come to be known as "prosecutor's fallacy."

The error stemmed from the prosecution expert wrongly conflating two very different mathematical probabilities: The probability that the crime scene evidence matched a person selected at random from the population and the probability that the defendant was guilty. In Brown's case, the court ruled that it was wrong for the state's expert to testify that there was a near 100% certainty of the defendant's guilt because that probability is based on a complex formula that takes into account factors such as the strength of other evidence against the suspect.

The court also concluded that the prosecution's DNA expert understated the chance that the DNA belonged to one of Brown's brothers. The victim had alternately identified Brown and one of his brothers as her attacker.

On Sunday, The Times published a report on another statistical issue in DNA cases -- the distorting effect of the database search in so-called cold hit cases, in which a defendant is first identified by trolling a large database of DNA profiles of convicted felons for a match to crime-scene evidence. Prosecutors and crime labs across the country have ignored recommendations by leading scientists to correct for the database effect, The Times reported, which could exaggerate the apparent strength of a DNA "match" by hundreds of thousands of times.

Unlike the database issue, the prosecutor's fallacy described in the 9th Circuit case is possible in all DNA cases. The case underscores what experts warn is the potential for distortion in DNA testimony, which relies on complex statistical evaluations.

joel.rubin@latimes.com

jason.felch@latimes.com