Torres won the light heavyweight title in 1965 by stopping Willie Pastrano at Madison Square Garden in New York. He made three title defenses before losing a close decision to Dick Tiger in 1966. He finished with a record of 41-3-1, with 29 knockouts.
The mayor of Ponce, where Torres will be buried Thursday, declared three days of mourning and ordered flags be flown at half-staff.
"Puerto Rico has lost a great Puerto Rican, a very valiant person who aside from being a great athlete, was a great human being," David Bernier, president of the U.S. territory's Olympic committee, told radio station WKAQ.
A native of Ponce, Torres was born May 3, 1936. He began fighting when he joined the U.S. Army boxing team as a teenager to avoid KP duty. He won a silver medal as a light middleweight at the 1956 Melbourne Olympics while competing for the U.S. He lost the title bout to Hungary's Laszlo Papp.
He debuted in 1958 with a first round knockout of George Hamilton in New York. He won his next 12 fights, 10 by knockout.
After retiring in 1969, he worked for several years as an aide to Paul O'Dwyer, then president of the New York City Council. He later worked for Andrew Stein, borough president of Manhattan. He also was Puerto Rico's official New York City representative.
Torres chaired the New York State Athletic Commission in the 1980s and served as supervisor for the World Boxing Organization.
In the 1990s, he fueled controversy while an aide to Mayor David N. Dinkins when he said that then-mayoral candidate Rudolph Giuliani appealed to the Ku Klux Klan.
Early in his career, he became friendly with several well-known writers, including Pete Hamill at the New York Post. With Hamill's help, Torres got a column in the Post that mostly dealt with Latino community affairs. Years later, Norman Mailer helped Torres write a book on Muhammad Ali, called "Sting Like a Bee." For several years, Torres would spar three rounds with the older Mailer.
Torres later wrote "Fire and Fear," a book on Mike Tyson, and was a columnist for New York Spanish-language newspaper El Diario La Prensa.
"Through his boxing, writing and speaking out on the important issues of our time, Jose was an inspiration to millions of people across the country and around the world," New York Gov. David A. Paterson said in a statement, noting that Torres was a New York Golden Gloves Champion in the 1950s.
He was inducted into the International Boxing Hall of Fame in 1997.