Traders dropped NetApp, salesforce.com and other tech companies for a second day running, sending their stocks down 2 percent or more. Whole Foods sank 19 percent after cutting its profit forecast.
Yellen told the Joint Economic Committee of Congress that a tough job market and weak inflation meant that the Fed will likely keep borrowing rates low for a "considerable time." As a result, she said, the economy still needed the Fed's help.
Yellen's comments appeared to ease concerns that the Fed was going to remove more support. The stock market had wandered lower in morning trading, then turned from a loss to a gain before the lunch hour.
"I think the market breathed a sigh of relief that she wasn't going to unveil something new," said Jeff Kleintop, chief market strategist at LPL Financial.
The Standard & Poor's 500 index gained 10.49 points, or 0.6 percent, to close at 1,878.21.
The Dow Jones industrial average climbed 117.52 points, or 0.7 percent, to 16,518.54.
The Nasdaq was the only major index to fall. It gave up 13.09 points, or 0.3 percent, to 4,067.67.
The S&P 500 index is within striking distance of its all-time closing high of 1,890 reached on April 2.
"Whenever you're near all-time highs you're going to see skittishness," said JJ Kinahan, chief strategist at TD Ameritrade. "In this market, the slightest news can change everything,"
Kinahan said that many investors are wondering whether the stock market is priced too high. The average stock trades at 16 times its earnings over the past year, according to S&P Capital IQ. That's slightly higher than the historical average. Some tech stocks, however, are valued much higher. Amazon's stock has lost 27 percent this year, but it's still trading at a lofty 465 times earnings.
"Many of these stocks have come down a lot, but you can't say they're cheap," Kleintop said.
High prices reflect expectations for higher earnings, and companies in the S&P 500 are on track to report that earnings increased 3 percent in the first quarter, according to S&P Capital IQ. The problem is, earnings growth is slowing down. In the previous quarter, earnings jumped nearly 8 percent.
And there are concerns about future profits. Of the companies that have provided forecasts for the second quarter, nearly nine out of 10 have warned of weaker earnings.
Whole Foods cut its profit outlook late Tuesday, saying it's facing increased competition as supermarkets, big-box stores and even online retailers step up their offerings of organic foods. It's the third time the grocery chain has reduced its profit forecast in the last six months. Whole Foods dropped $9.02, or 19 percent, to $38.93.
Among Internet stocks, NetApp, a data management and storage company, fell $1.28, or 4 percent, to $33.70 and salesforce.com lost $1.35, or 3 percent, to $50.43.
Just two of the eight sectors in the S&P 500 fell, information technology and consumer discretionary companies. Gainers included utilities, which rose the most, 1.6 percent. That's a sign investors are still cautious. Investors tend to favor less volatile, high-dividend stocks like power companies when they want to play it safe. Utilities are by far the best-performing sector in the market so far this year, up 13.8 percent.
In other markets, crude oil rose $1.27 to settle at $100.77 a barrel. Gold dropped $19.70 to $1,288.90 an ounce. U.S. government bonds barely moved. The yield on the 10-year Treasury note ended the trading day at 2.59 percent, unchanged from late Tuesday.
Among other companies making big moves:
— Mondelez surged following news that it will combine its coffee business with D.E. Master Blenders to form a new company, Jacobs Douwe Egberts. The new company will sell Gevalia, Tassimo and Jacobs, among other coffee brands. Mondelez gained $2.88, or 8 percent, to $38.10.
— Electronic Arts jumped $5.90, or 21 percent, to $33.95 after the video-game maker turned in stronger results late Tuesday. The maker of "The Sims" and `'Madden NFL" reported higher profits and revenue than Wall Street expected and forecast stronger earnings over the next year. Electronic Arts has soared 45 percent so far this year.
— Molson Coors Brewing reported better results than analysts expected on Wednesday. Quarterly earnings rose thanks to a payment it received from Modelo for a joint venture that ended early. Molson's stock rose $2.18, or 4 percent, to $61.93.