Johansson died at a nursing home in Kungsbacka on the Swedish west coast, his daughter Maria Gregner told the Associated Press.
"He died peacefully in his sleep about 20 minutes before midnight," Gregner said. "It was his illness ... it was time."
Johansson was diagnosed with Alzheimer's disease and dementia more than 10 years ago when he lived in Stockholm. He spent the rest of his life in Kungsbacka, only a few miles from the house where he grew up.
Known as "Ingo" to Swedes, Johansson knocked out Patterson in the third round at New York's Yankee Stadium on June 26, 1959, to win the heavyweight title. He floored the American seven times before referee Ruby Goldstein stopped the fight 2:03 into the round.
Back home, hundreds of thousands of Swedes listened to the live radio broadcast at 3 a.m. as Johansson became only the fifth heavyweight champion born outside the United States. Swedish newspapers printed extra editions with Ingo on the cover.
"What he did was the biggest feat ever in Swedish sporting history," his longtime friend Stig Caldeborn said. It earned Johansson The Associated Press' Male Athlete of the Year honors in 1959, only the second Swede to win the award.
"During his time he was regarded as one of Sweden's greatest athletes, and he still is," Swedish Prime Minister Fredrik Reinfeldt was quoted as saying by news agency TT.
In 2000, the Swedish Sports Academy selected Johansson as Sweden's third best athlete in the 20th century behind tennis star Bjorn Borg and alpine ski legend Ingemar Stenmark.
Patterson avenged the upset loss a year later in the rematch in New York, knocking Johansson out in the fifth round. In March 1961, the Swede floored Patterson twice in Miami before being knocked out in the sixth round of the rubber match. Patterson died in 2006.
Johansson had four more fights -- all wins, one of them a knockout of England's Dick Richardson for the European title in 1962 -- before retiring the following year. He finished his career with a 26-2 record, including 17 knockouts.
A well-schooled upright boxer, Johansson had a good jab that helped set up a tremendous knockout right hand dubbed "Ingo's Bingo" and the "Hammer of Thor."
Johansson went 61-10 with 31 KOs as a decorated amateur. His biggest disappointment came at the 1952 Olympics in Helsinki, Finland, where he was disqualified in the heavyweight final for not giving his best.
Johansson always claimed that he backed away in that fight in an attempt to lure his American opponent Ed Sanders into his right-hand counter. The Swede eventually received his silver medal 30 years later from the International Olympic Committee.
Johansson became a businessman after finishing his boxing career. He owned a fishing boat named "Ingo" and a bar called "Ingo's" in Goteborg, Sweden's second biggest city.
Johansson later moved to Florida, where he operated a hotel at Pompano Beach and started playing golf. He also completed the Stockholm Marathon before hundreds of thousands of spectators in 1985.
Johansson was married and divorced twice, and is survived by six children. Funeral arrangements were not immediately announced.
Associated Press Writer Karl Ritter contributed to this report.