The university said James died at his home from the effects of pancreatic cancer. He had been receiving treatment for the disease since late September.
The Huskies' leader from 1975 to 1992, he won more games than any other coach in the school's 128-year football history.
He left in 1993 to protest what he felt were unfair sanctions levied against the team by the Pacific-10 conference. The penalties, which included a two-year ban on post-season bowl games, were the conference's response to allegedly improper loans to and compensation of athletes by team boosters.
"I have decided I can no longer coach in a conference that treats its players and coaches so unfairly," the coach affectionately known as "the Dawgfather" wrote in a letter of resignation he submitted just two weeks before the opening of the 1993 football season.
James, who was head coach at Kent State University in Ohio before accepting his UW position, led the Huskies to a 12-0 season in 1991.
With the most dominant defense in the country, Washington beat Michigan that year in the Rose Bowl. The Associated Press media poll gave the University of Miami — James' alma mater — the national championship, while the coaches voted in favor of Washington in their poll.
"His accomplishments as a football coach stand alone, but what made him truly special is the quality of man he was away from the game," current Washington Coach Steve Sarkisian said.
After his resignation, he was a regular visitor at team practices and served on a committee that helped redesign Husky Stadium. He gave an annual preseason speech to the football squad, delivering his most recent talk last August.
Born Dec. 31, 1932, James was the son of a bricklayer in Massillon, Ohio. He was a standout football player in high school, accepting a scholarship from Miami and setting five passing records there before graduating in 1954.
After serving in the Army, he received a master's degree in education from the University of Kansas in 1957.
He held coaching positions at Florida State, Michigan and Colorado before taking the job of head coach at Kent State in 1971.
In Seattle, he was credited with reviving a flagging program. The Huskies had won just six games in the two years before James arrived; two years later, they beat Michigan in the Rose Bowl. It was the first of James' six Rose Bowl trips.
He was inducted into the College Football Hall of Fame in 1997.
James is survived by his wife of 61 years, Carol: their children Jeff, Jill and Jeni; and 10 grandchildren.