Love and Madness on the Frontier : THE YELLOWSTONE <i> by Winfred Blevins (Bantam Books: $3.95, paper; 304 pp.; 0-553-27401-5) </i> : THE HOMESMAN <i> by Glendon Swarthout (Weidenfeld & Nicolson: $17.95; 243 pp.; 1-55584-235-6) </i>

When “Lonesome Dove” was awarded a Pulitzer Prize in 1986, an entire literary sub-genre gained deserved recognition. The “frontier novel” is a hybrid; it has all the density of character and situation of a historical novel, without the pretentiousness; and it has the immediacy and sense of danger that pervades the Western, without the moral simplicity. More than anything else, the frontier novel is about people and places on the edge of things.

Two new and quite different releases--Winfred Blevins’ “The Yellowstone” and Glendon Swarthout’s “The Homesman”--typify the genre.

“The Yellowstone” is frontier novel as saga. In 1841, 21-year-old Robert Burns Maclean is separated from his fellow trappers. Mac discovers the wonders of the Yellowstone River, its country and its inhabitants. Mac’s love affair with the Yellowstone will dominate his life, affecting his choice of wives, work and home.

The Yellowstone is a raw place, and this is reflected in its people. There is rape and murder, polygamy and love, frontier violence and frontier justice. And injustice.

Blevins’ affection for his characters--villains and heroes alike--is evident. He is knowledgeable about both the natives and the rough invaders who walk the edge of encounter and conflict. The short vignettes describing frontier life are enjoyable and sometimes startling; Cheyenne sexual and courting mores prove fascinating.

But when Blevins attempts to describe the Yellowstone, assertive adjectives and phrases substitute for subtlety and evocation. In one brief section we are offered: “enchanted,” “enchantment,” “fairy tale,” “miraculous,” “exceedingly beautiful,” “high ethereal peaks,” “unimaginably blue,” “marvelous lake,” “strange hot springs,” “majestic,” “verdant country” and many “miracles.” There is more poetry in the simplicity of a trail guide.

The novel itself is episodic and lacks dramatic tension. Even scenes that should work--a father waiting to be ambushed by his son; a seduction scene--are weakened by lack of development: We have not been allowed to see the father’s relationship with the son; we know nothing of the man’s sexual history until after the seduction.

“The Yellowstone” should be a powerful story, but the author’s voice is unconvincing. Incidents become incidental. We are left too distant from landscape, event or character to be moved.

There is no distancing in “The Homesman.” From its very first words, Glendon Swarthout’s frontier novel is totally involving. It is nothing less than a study of the human spirit.

In an unnamed territory 90 miles from Wamego, over a long, hard winter, four women have gone crazy. A “homesman” is needed to take them to Hebron, where the Ladies Aid Society of the Methodist Church will then help send them “home.”

Mary Bee Cuddy, a spinster homesteader, is trapped by circumstances into volunteering for the physically and emotionally arduous task. She enlists the aid of Briggs, a claim-jumper she saved from hanging.

What follows is a dangerous journey into the soul, an exploration of the relationships of men and women to each other, to their environments and--ultimately and most devastatingly--to themselves.

Swarthout writes quietly, slipping in and out of the hearts of his characters. The turns-of-phrase are satisfying; they convey both wryness and a sense of the period (“Wamego . . . had more lawyers than human residents”). We learn what it is to live on a frontier, to be exposed to circumstances that must ravage the psyche.

In one scene, we encounter Theoline Belknap. She is in labor with her fifth child, a child unwanted and unavoidable. It is the middle of a “winter of damnation.” Her husband has gone for food. Alone except for her two young daughters, separated from them by a hung blanket, Theoline prepares to deliver her own baby:

“Reaching, she took hold of the two ropes tied to the bedposts, one in each hand, and pulled on them as she pushed with her lower parts, pulling and pushing and screaming now and her girls screaming.

“The baby presented itself head-first.

“She let go of the ropes and freed herself of it and saw that it was perfect and a girl.

“She lifted it by its slippery legs and shook it until it began to cry.

“She laid it between her legs and ran a forefinger around inside of its mouth to rid it of mucus and to make sure its tongue was straight forward, not back, so it would not strangle. . . . Then she wiped the tiny thing off tenderly with bedding, nestled it beside her, placed the dishpan between her legs, and fell back against the pillow, dead tired.

“The girls were silent now, but she thought she could almost hear hearts beating behind the quilt. . . .

“As soon as she was strong enough, she . . . went barefoot . . . to the outhouse, opened the door, stepped inside, and pushed the naked baby down the hole head-first.”

“The Homesman” is impressive, occasionally shattering, always convincing. With simple language and understatement, Swarthout conveys the loneliness and terror of people trying to survive the unsurvivable. To read “The Homesman” is to be on the edge of things.