It’s a Vivid Tale of Youth--Too Bad the Writer Intrudes : LUCKY TOWN <i> by James Brown</i> . Harcourt Brace; $22.95, 287 pages

Important stuff happens to us when we’re young, but we have to grow up before we can figure it out--by which time we’ve forgotten much of it, distorted much more and, at best, lost touch with its power and immediacy.

This poses a problem that every memoirist and every novelist who writes about youthful experience has to solve. Should the writer speak as an adult, at the risk of drowning out the voice of the child? Pretend to be a child, at the risk of losing the perspective of the adult? Or shift back and forth, hoping to strike an optimum balance?

There’s no right answer. Good books--and bad--have been written all three ways. In “The Catcher in the Rye,” J.D. Salinger seems to vanish into Holden Caulfield; the adult insights he expresses seem to come to us subliminally. In “This Boy’s Life,” Tobias Wolff balances his youthful fear of an abusive stepfather with a grown-up view in which the man is pitiable as well as mean.

In “Lucky Town,” James Brown speaks mainly as an adult: 36-year-old Bobby Barlow, looking back two decades on his love-hate relationship with his handsome, charming dad, Floyd, whose erratic lunges at the American Dream involve romantic obsession, gambling and armed robbery.

Bobby’s mother has disappeared. He has been raised by Floyd, a carpenter whose failings seem no worse than immaturity and impatience (driving through an affluent neighborhood, he mutters, “ I should be in that nice warm house. I should be behind the wheel of that Dodge”) until he meets his femme fatale, Melinda Johnson, and commits his first crime.

Floyd spends three years in an Oregon prison, emerging somewhat hardened and even more impulsive. Bobby senses this, yet immediately leaves his foster family in Portland to be with him--although Floyd’s only plan is to track down Melinda, a former hooker and sometime law student now married to a brutal Seattle cop named Bo.

Brown makes us understand why Bobby chooses his father. And he does it the hard way, by giving Floyd such a mixture of machismo and winsomeness, violence and vulnerability, good intentions and bad judgment that few boys could help loving him--even though, Bobby soon realizes, “He was the child. I was the parent.”

The minor characters are equally vivid. Brown’s prose in direct scenes is crisp--he has an especially nice touch with background sounds--and he can sum up a relationship in a few lines of dialogue. When Melinda leaves Bo and runs off to Las Vegas with Floyd and Bobby in a stolen car, Bobby asks her: “I know what my father sees in you. What I don’t get is what you see in him.”

“Like he does things. Like he’s going places. . . . He knows how to treat a woman, too, plenty better than most of the jerks I’ve known. . . .”

“I know one thing. We wouldn’t be in this mess right now if it wasn’t for you.”

“I hate to break your heart,” Melinda tells him. “But let’s get something straight right now. If anyone goes it’ll be you.”

She’s right, although it’s a long time before Floyd is desperate enough to prove it. In Las Vegas--”Lucky Town”--he prospers for a while at the craps tables, aided by Melinda, who lands a job as a dealer. Even after Floyd tries to make a killing with loaded dice and is beaten up by casino hoods, Bobby can pity him for his knocked-out tooth and cut-off finger, the two other fingers lost in a sawmill accident, his doomed passion: “The years hadn’t been kind.”

It takes a whole string of selfish and stupid acts for Floyd to alienate his son, but he finally does it. We can understand Bobby’s alienation, just as we understood the love. What we can’t quite grasp is how this smart-aleck, pot-smoking “hippie” teen-ager has become the distant, slightly stuffy and long-winded adult who tells the story.

Bobby seems to be saying that he wised up to his father’s flaws and simply walked away. If this were a lie--if we knew Brown intended for us to see the adult Bobby as wounded and in denial, reliving a past he can’t quite shake--then the novel would work in the past and the present. But as it is, the adult Bobby speculates about the past without adding anything to it; his voice is only a distraction to the compelling story being told.