Mistrial Ends New Yorker Libel Case : Courts: Jury deadlocks on the question of damages. However, they conclude that writer defamed psychoanalyst.

A psychoanalyst’s libel case against the New Yorker magazine and writer Janet Malcolm ended in a mistrial Thursday, but it will take a judge’s ruling to define what that conclusion means.

On their fourth day of deliberations, jurors delivered a partial verdict concluding that Malcolm had libeled Jeffrey M. Masson with two quotations in a New Yorker article.

In a separate finding, the eight federal court jurors absolved the magazine of blame, concluding that Malcolm was an independent contractor--not a New Yorker employee--when she wrote her profile of Masson in 1983.

The verdict is in doubt because the jury could not agree on how much money--if any--Masson deserved. He had asked for $7.5 million, arguing that the magazine portrait made him unemployable and a laughingstock in academic circles.

When the jurors said Thursday that they were hopelessly deadlocked on monetary damages, U.S. District Judge Eugene F. Lynch dismissed them and said he will hold a hearing within several weeks to decide several murky questions left in the trial’s wake:

* Will the entire case be retried?

* Will the libel verdict stand and only the damages portion be re-argued?

* If that occurs, will the New Yorker be excused as a target of the lawsuit?

Masson’s San Francisco attorney, Charles O. Morgan Jr., said he would probably press for a retrial of the entire case. He favors that strategy, legal experts said, because it would give a second jury the chance to blame the New Yorker for the libelous quotes--and probably increase the amount of money Masson could collect if he wins.

Not surprisingly, Malcolm’s lawyer said he also favors a second trial of all issues. “As much as I wish it were over, I think it should all be tried all over again,” Gary Bostwick of Santa Monica said.

As for the New Yorker, its lawyers said they will argue that the publication has been exonerated by the jury and should no longer be part of the case. But the magazine’s editor, Tina Brown, said that she “continues to stand by the integrity of (Malcolm’s) work and will continue to publish it with pride.”

Despite the legal uncertainties, Masson, 52, said he feels vindicated and heartened that “eight people said: ‘You didn’t say those things. She fabricated them.’ ”

Malcolm, 58, left the courtroom looking shaken and would not comment. Bostwick said his client feels very badly but added that she had recently recited a line from “Gone With the Wind”--a line suggesting that she is upbeat about the future.

“She reminded me . . . of what Rhett Butler had said, that people with character don’t need reputations,” Bostwick said. “She knows she told the truth. And it doesn’t necessarily (upset her) to hear what eight other people thought.”

Thursday’s odd finale caps one of the most riveting libel battles in recent history. Marked by an intriguing cast of brainy characters and loads of titillating testimony, the trial packed Lynch’s courtroom for three weeks and enthralled lawyers and literati on both coasts.

Some spectators were lured by the deftness of the opposing attorneys--Morgan, an avuncular sort whose folksy manner masked his nimble handling of witnesses, and Bostwick, who displayed a dry wit and go-for-the-jugular style.

Others found that the trial provided a rare peek inside the cloistered world of the New Yorker, exposing the magazine’s inner workings and the methods of one of its veteran writers.

Most people were drawn by the warring litigants, whose unpredictable, self-exploratory testimony at times gave the trial the feel of a big group therapy session.

During his turn on the witness stand, the loquacious Masson discussed everything from his sex-obsessed youth--he once said he slept with 1,000 women--to his conversion to vegetarianism.

Malcolm, a prim witness who clenched her fists as she testified, raised eyebrows with her admission that she tidies up her subjects’ sloppy speech and her furious denial that she once sought to interview Masson on a nude beach.

The legal feud began in 1984, shortly after Malcolm wrote her two-part series about Masson, his iconoclastic views on Sigmund Freud and his being fired as curator of the Freud Archives.

Masson agreed to talk with Malcolm in the hope that her magazine article would win converts to his views and perhaps rescue him from professional exile. However, much of the 48,500-word piece cast him as a braggart with an insatiable libido. Masson sued.

The imbroglio has stretched over nine years--and reached the U.S. Supreme Court in 1991. In its decision, the high court reinstated Masson’s claim and refined the nation’s libel law, ruling that plaintiffs must prove that their fabricated quotes caused a material change in their meaning, were defamatory, and that the writer knew they were false or used them with reckless disregard for their truth or falsity.

Masson said he was libeled by five quotations. Jurors agreed that Malcolm had fabricated or distorted those five comments. But only two were defamatory and known by Malcolm to be false, and thus libelous, the jury concluded.

One quote is on tape recordings of Malcolm’s interviews, but Masson argued that the journalist altered its meaning by deleting preceding phrases. The other quote appears in typewritten notes that Malcolm said she produced from rough originals, which she cannot find.

Only one juror, Patricia Brooks, would comment Thursday. Brooks said that the jury regretted not being able to reach an agreement on damages, but “that’s the way it goes.”

Asked whether Masson’s promiscuity--a central theme in the trial--had affected the deliberations, Brooks said it was not a factor: “Everybody’s got their own lifestyle.”