When he was honorably discharged from the military, he decided to teach himself to sew.
"He was a drag queen," explained his longtime friend Cleve Jones. "He sewed gowns and costumes."
Soon he began sewing flags for gay and antiwar street marches.
In 1978, he designed and sewed the banner that would become a symbol for LGBTQ rights around the world: the rainbow flag.
Baker died at 65 in his sleep at his New York home, said Jones, a longtime gay rights and AIDS activist. He did not know the cause of death, but said Baker had suffered a massive stroke several years ago and had to teach himself to sew again.
He said Baker, his best friend for 45 years, was a creative, big-hearted force of nature.
"He was crazy," Jones said. "He was still a hippie at heart, part of the psychedelic generation."
In the mid-1970s, gay activists in the Castro District of San Francisco — including Jones, Baker and Harvey Milk — were trying to come up with a symbol for their movement.
Baker didn't like two symbols they were using: The Greek letter lambda seemed too obscure, and the pink triangle that the Nazis had forced gay prisoners in their concentration camps to wear was too dark.
"Gilbert's idea was to create something that reflects the diversity of the LGBT community," Jones said.
Milk, the first openly gay San Francisco supervisor, urged Baker to come up with a symbol of hope, according to Baker's online bio.
On the city's Gay Freedom Day Parade in 1978, Milk rode under the first rainbow flags sewn by Baker, and dyed by him and Jones. Milk was assassinated five months later, along with the then-mayor, George Moscone.
The first flags had eight color stripes, but Baker removed pink and turquoise because those fabrics were more expensive. The six simple panels of red, orange, yellow, green, blue and purple would become a universal beacon of pride for gay and transgender communities.
In 1979, Baker went to work at Paramount Flag Co.in San Francisco and created "flamboyant window displays," according to his online biography. Dianne Feinstein, now a U.S. senator, saw them and commissioned him to design flags for her first inaugural as mayor.
Baker liked to joke that he was the "gay Betsy Ross."
He moved to New York in 1994 and created a mile-long rainbow flag to commemorate the 25th anniversary of the Stonewall Riots, widely viewed as the start of the gay liberation movement.
He did fine art silk screens on linen celebrating the flag, and worked with photographers to document it. He held exhibitions in Rome, New York and San Francisco.
In 2003, for the 25th anniversary of the symbol, he built his largest flag yet — stretching across Key West from the Gulf of Mexico to the Florida Strait. Five years later, he returned to San Francisco to re-create the 1970s banners for the movie "Milk."
"Gilbert's presence in the world will never be forgotten," Jerome Goldstein, a close friend, told the San Francisco Chronicle.
"HIV disease may be forgotten, all of the ugly political and religious fights that we've had, they may be forgotten," Goldstein said. "But the rainbow flag will endure forever."