Flournoy's decision underscores the difficulty President Obama may face in finding a candidate to take the helm at the Pentagon late in his second term and as the administration faces intense criticism of its handling of crises in the Middle East and elsewhere in the world.
Other candidates being considered include Homeland Security Secretary Jeh Johnson, who previously served as the Pentagon's general counsel, according to several people close to the process. Johnson is highly regarded by the West Wing, particularly after he oversaw the months-long process to identify the immigration executive actions Obama announced last week.
But given Republicans' staunch opposition to those actions, tapping Johnson for the Pentagon post risks turning his confirmation hearing into a fierce debate on immigration. The president would also need to fill the top job at Homeland Security again just as that department is implementing the immigration actions.
Two Pentagon veterans are also seen as contenders for secretary of Defense: former Deputy Defense Secretary Ashton Carter and Robert Work, who currently holds the No. 2 job at the Pentagon.
Still, Flournoy's decision to withdraw removes a candidate who was seen as being at the top of Obama's list. Flournoy, who was also considered for Defense secretary in 2012, would have been the first woman to head the Pentagon.
Flournoy served as a top Pentagon official during Obama's first term, then returned to the Center for a New American Security, the think tank she cofounded. She sent a letter to the think tank's board Tuesday saying she had asked the president to take her out of consideration for the Pentagon job, citing family issues.
A person close to Flournoy said she had concerns about the job, including whether she would be given more latitude than Hagel in policymaking. Pentagon officials have long griped about White House micromanagement, and Hagel was largely seen as someone who would acquiesce to the West Wing.
Hagel submitted his resignation Monday under pressure from the president. The former Republican senator struggled to break through Obama's insular inner circle of foreign policy advisors. And despite the White House's desire to have a less independent Pentagon chief, officials privately complained about Hagel's lack of engagement in national security deliberations.
The White House would not comment on the process to nominate Hagel's successor.
"The president is considering a number of well-qualified candidates; however, I don't have any personnel announcements at this time," National Security Council spokeswoman Bernadette Meehan said.
People familiar with the process insisted on anonymity because they were not authorized to discuss the matter by name. Flournoy's decision was first reported by Foreign Policy magazine.