House Speaker John A. Boehner (R-Ohio) had invited Netanyahu without telling the White House. The Israeli leader is scheduled to speak next month, about two weeks before Israel's elections on March 17.
Relations between President Obama and Netanyahu have been strained for years.
Obama has said he will not meet with Netanyahu because his visit is too close to the Israeli elections and could create the appearance of U.S. interference. Vice President Joe Biden, who as president of the Senate would usually preside over such a session, has said he will be traveling and unable to attend.
On Tuesday, more lawmakers backed away.
Sen. Brian Schatz of Hawaii, a Jewish Democrat, said the scheduled event “does more harm than good to the bipartisan U.S.-Israel alliance.”
“The U.S.-Israel relationship is too important to be overshadowed by partisan politics,” he said.
Sen. Patrick J. Leahy of Vermont, the most senior Democrat in the Senate, criticized Republican leaders for the “unfortunate way [they] have unilaterally arranged this, and then heavily politicized it.”
“They have orchestrated a tawdry and high-handed stunt that has embarrassed not only Israel but the Congress itself,” he said in announcing he would not attend the speech.
Top Israeli officials worked the halls of the U.S. Capitol last week to gauge the level of concern, particularly among Democrats, over the speech controversy.
The speaker of Israel's Knesset met separately with House Speaker John A. Boehner (R-Ohio) and House Minority Leader Nancy Pelosi (D-San Francisco), and the Israeli ambassador to the United States met with a small group of pro-Israel Democrats who expressed concern at how the planned visit has devolved into a partisan fight.
Pelosi told reporters later that she hoped the speech “doesn’t take place,” and would not commit to attending.
“Some staunch supporters of Israel have commented it's outrageous -- and they're supporters of Netanyahu -- that our floor of the House would be exploited in that way for a political purpose in Israel and in the United States,” she said.
“The way this conversation is taking so much energy and is really stressful, is really beneath the dignity of the challenge that we have -- stopping the proliferation of weapons of mass destruction.”
Pelosi also raised the possibility that Netanyahu might look for a way out.
“You never know. Things happen in people's schedules,” she said.
Rep. Steve Israel (D-N.Y.), who hosted the meeting between Israeli Ambassador Ron Dermer and the Democrats, said the lawmakers were seeking to "defuse" the political tension the invitation has prompted.
Though he declined to comment extensively on the substance of the conversation, Israel said lawmakers discussed the timing of Netanyahu's speech and whether his visit would undermine the traditional bipartisan nature of U.S. support for Israel.
“The bottom line was the concern that style has overtaken substance, and we need to get back to the substance of the relationship between the United States and Israel, and not define that relationship by one poorly rolled-out invitation to the prime minister to address Congress,” Israel said.
Dermer told the lawmakers that, as in previous speeches to Congress, Netanyahu would discuss how such bipartisan support was the "most important strategic asset" Israel has. Dermer also said he would relate the Democrats' concerns to the prime minister.
“I don’t know whether the prime minister would consider not coming, but I think that there’s some consideration of whether that should be considered,” Israel said. “If they can find ways to relieve some of the concerns over timing, then that might be better.”
Separately, a group of Democrats including Reps. Maxine Waters of Los Angeles and Keith Ellison of Minnesota -- the first Muslim elected to Congress -- have been collecting signatures for a letter to Boehner asking him to postpone the speech until after the multinational nuclear talks with Iran are resolved, saying the invitation "has the potential to harm U.S. foreign policy." So far 20 Democrats have signed on.
"Aside from being improper, this places Israel, a close and valued ally, in the middle of a policy debate between Congress and the White House. We should not turn our diplomatic friendship into a partisan issue," the letter reads. "When the Israeli prime minister visits us outside the specter of partisan politics, we will be delighted and honored to greet him or her on the floor of the House."