"We have released, both under my administration and previous administrations, a large number of former Taliban fighters, some of whom will return to the battlefield," he said. "But by definition, you don't do prisoner exchanges with your friends, you do 'em with your enemies."
He said other tough choices lie ahead as the U.S. extricates its forces from Afghanistan over the next two years.
"And I've been very clear about the fact the over time, we're going to have to whittle away at the number of prisoners who were in Guantanamo as part of this transition out of the war in Afghanistan."
About 150 detainees remain in the U.S. military prison at Guantanamo Bay, Cuba, most for more than a dozen years. None has been brought to trial. The facility, set up after the Sept. 11, 2001, terrorist attacks, held more than 750 detainees at one point.
Under international law, countries typically release or repatriate prisoners of war at the end of hostilities.
An administration official, who spoke on condition of anonymity to discuss internal deliberations, said the White House doesn't see the expected withdrawal of all U.S. troops from Afghanistan by the end of 2016 as a legal tripwire, however.
"So, is this part of that Guantanamo drawdown?" Williams asked.
"But by definition, if we, in fact, are ending a war," he said, "then there's going to be a process in which some of those individuals are going to be released."