Holder was anathema to Republicans, who held him in contempt of Congress in 2012 for refusing to hand over documents in the "Fast and Furious" gun-trafficking scandal. On Thursday, Rep. Darrell Issa (R-Vista), chairman of the House Oversight and Government Reform Committee, called Holder "the most divisive attorney general in history."
But Holder's relationship with his natural constituency, liberal Democratic lawyers, has been complex. Many were deeply disappointed by his first four years. Even close confidants concede that Holder was so preoccupied with controversy and political scandals that he was unable to initiate the kinds of changes in civil rights and civil liberties that were expected of the nation's first African American attorney general.
Holder surprised friends by deciding to stay on for the beginning of the second term, apparently with the promise from his personal friend in the Oval Office that with reelection behind them Holder would be free to pursue the liberal civil rights agenda he clearly had wanted from the beginning.
In the last 18 months, he has rallied the Justice Department and even Congress around reforms throughout the criminal justice system, from early release of nonviolent prisoners to easing mandatory prison sentences, which fall heaviest on minorities.
He has been a strong advocate of voting rights, fighting to resuscitate the 1965 Voting Rights Act through new legal interpretations after a devastating blow from the Supreme Court. He has fought for renewed voting rights for felons.
Gay rights groups were also unreserved in their praise for Holder, who worked aggressively to interpret last year's Supreme Court decisions on gay marriage in the broadest possible way so the maximum number of gay couples would receive new federal benefits.
But other liberals were disappointed by Holder's record on privacy and national security and by what they said was his failure to prosecute Wall Street titans responsible for the recent 2008 financial collapse.
They also remain unhappy with his decision not to prosecute Bush administration officials for authorizing the use of torture after 9/11.
Holder came under fire for going after reporters' sources in national security leaks, alienating 1st Amendment advocates.
Only recently has the Justice Department gotten settlements from some major banks for their role in the financial meltdown, and no high-profile Wall Street figures have been held criminally liable.
"He has done amazing things on race and criminal justice issues. That will be Holder's legacy," said the advocate. But on other issues, he has fallen short, he said.