BOOK REVIEW : Mystery of the Sisters in the Tower : THE DARK SISTER <i> by Rebecca Goldstein</i> ; Viking; $19.95; 260 pages

“The Dark Sister” is at once an erudite and frustrating novel. Rebecca Goldstein, a professor of philosophy at Rutgers, has packed her third novel with abundant literary and philosophical references--fascinating ideas.

She acknowledges “borrowing” liberally from many sources, including the writings of Thomas Hardy, Wilkie Collins, Edgar Allan Poe and especially--most especially--those formidable siblings Henry, William and Alice James, who figure prominently as characters in “The Dark Sister.”

Initially, it seems like a wonderful idea: a novel featuring Henry and William James, novelist and philosopher respectively, two of the most brilliant figures in modern letters, and their younger sister, Alice, a lifelong invalid who wavered between self-effacement and intellectual fervor.

Part of the problem lies in the novel’s complex structure. The author attempts to tell two stories at once, creating a novel within a novel, intertwining a contemporary story with a Victorian one. Switching back and forth, a reader hears on one hand a most contemporary voice--that of Hedda Dunkele, the novel’s main character--and is confronted, only a few pages later, by rather turgid, very formal Victorian prose.

The story, or stories, goes like this: Hedda, an author of feminist novels featuring what she calls AJWs--Angry Jewish Women--has embarked on a new and radically different sort of book. She is writing a Victorian novel which is doubly Jamesian: Henry is dictating the style, and William is emerging as a major protagonist.

Hedda’s novel, called “The Dark Sister,” features two New England sisters, Alice and Vivianna, reclusive spinsters who dwell in a gloomy tower and who, in their discordant feelings, bear a striking resemblance to Hedda and her own younger sister, Stella.

Both sets of sisters--the fictional and the real--are seriously at odds with each other, yet are bound by “bloody history.” The immutable rivalry between sisters forms a theme in the book, although the sources of such rivalry feel curiously unexplored except in terms of absent parents. The younger sisters, Stella and Vivianna, are deemed to be psychologically unsound, while Hedda and her fictional counterpart are at first presented as sane, only to later be revealed as delusional.

In Hedda’s novel, William James is called upon by his friend, Dr. Sloper, to examine Vivianna, who seems to suffer from clinical hysteria.

“I know your predilection for the more exotic specimens in the garden of abnormal personality,” Dr. Sloper tells James, “most especially if the rare bloom should carry a faintly metaphysical tint to it.”

Vivianna has that metaphysical tint. In truth, upon meeting her, William James finds Vivianna charming, if morbid. By day, she sleeps. She spends her nights looking heavenward in an attempt to make astronomical discoveries. Yet her sadness is palpable. What, James wonders, is the secret behind this sadness? In attempting to answer this question, he consults his brother, Henry, in letters and muses on the tragic aspects of his own sister Alice’s life.

As William James becomes more involved with the sisters, he also deepens his engagement with the supernatural, attending seances conducted by an oracle named Mrs. Piper, which prove to be some of the most lively passages in the book.

In fact, one of the strengths of “The Dark Sister” is its presentation of the figure of William James. Goldstein brings to her subject a sense of the man’s quickness and audacity of mind. James is a vivid, clever fellow, a man “not given to easy enthusiasms or breathless alacrity.” In time, he unravels the mystery behind the story of the sisters in the tower. In the process, the reader is treated to insights into the man and his times.

William James, the pragmatist who championed free will (his first act of free will was to believe in free will), comes off as the most compelling figure in the novel. One can imagine the entire novel having been composed of a fictionalized story of James, his relations with his brother, his wife, Dr. Sloper and his daughter, Catherine, and the “phantasmagoric” life he sought to contact through mediums. The result might have been a more cohesive book.

By contrast, the chapters featuring Hedda feel unfocused. The line is often blurred between the author’s voice and Hedda’s, as though an actress were at one moment delivering lines in a play and the next simply chatting with her audience.

Throughout the novel, Hedda quotes lines from a poem by Adrienne Rich, “A Woman in the shape of a monster, a monster in the shape of a woman, that shape am I, potentially.”

It’s meant to be a feminist statement. But Hedda doesn’t seem like a monster so much as a woman who’s lost her identity, a neurotic novelist, burdened by petty jealousies and anger directed in equal measure toward her sister, literary critics, piggish men and insensitive readers.

The effect is unpleasant. By mixing disparate narratives so freely, Goldstein has watered much of the richness in her story out of existence.

NEXT: Carolyn See reviews “Zelda Fitzgerald: The Collected Writings,” edited by Matthew J. Bruccoli (Scribner’s).