His wife, Patricia Ali, said he died at Manhattan's Bellevue Hospital of a blood clot in his lung.
Ali joined Coltrane's group in the mid-1960s during the saxophonist's period of avant-garde jazz experimentation. When Coltrane decided to use two drummers in a concert at the Village Gate in November 1965, he chose Ali to back up drummer Elvin Jones.
Ali recorded with both men on the 1965 album "Meditations" and accompanied Coltrane alone on "Interstellar Space," recorded shortly before Coltrane died of cancer in 1967. On that album, Ali improvised with irregular meters.
After Coltrane died, Ali toured Europe before returning to New York to play and record there. He opened the jazz club Ali's Alley in 1973 and launched the Survival Records label, which he maintained until his death. The club closed in 1979.
Ali was born Robert Patterson on July 1, 1933, to a musical family in Philadelphia. He began playing drums in R&B groups there before experimenting with free jazz.
Ali toured with saxophonist Sonny Rollins before moving in 1963 to New York, where he found greater acceptance for his improvisational style.
Over a career that spanned more than four decades, he performed with artists including Don Cherry, Albert Ayler, Alice Coltrane and Archie Shepp. In recent years he formed The Rashied Ali Quintet, and this year he released a "Live In Europe" album with the group.
"He was at the top of his game until his last day," his wife said.
Last weekend Ali was on the bill at the Newport Jazz Festival with a free-jazz trio called By Any Means along with alto saxophonist Charles Gayle and bassist William Parker.