Federer, who played his first tournament in nearly four weeks, made too many mistakes in a 6-4, 7-6 (5) loss to Gael Monfils after Wawrinka — the defending champion — was routed, 6-1, 6-2, by Grigor Dimitrov.
“It's the very beginning of the clay-court season. It's going to take me some time to feel 100 percent comfortable,” the second-ranked Federer said. “Not many guys are perfect this week.”
Nadal might not be at his best yet, but his fighting spirit remains intact. The eight-time Monte Carlo champion, who arrived with a big question mark surrounding his form, came out on top of a tense match with the big-serving Isner to reach the quarterfinals with a 7-6 (6), 4-6, 6-3 win.
Top-ranked Novak Djokovic enjoyed another stroll and extended his winning streak to 14 matches, defeating Andreas Haider-Maurer 6-4, 6-0 to set-up a quarterfinal match against Marin Cilic.
Federer has changed his schedule this season to allow more time for training and was not expecting to peak early on clay. He will now return to Switzerland to do some more practice before tournaments in Istanbul and Madrid.
At 33, Federer's main ambition is to add more major titles to his glittering resume. His quest for a second French Open begins on May 24 and he believes he has enough time to make the proper adjustments.
“It was a good week for me anyways to come here and practice with the best, play a couple of matches,” said Federer, a four-time runner-up at the Country Club. “At least it gives me some information, if I'm trying to be a bit positive right now.”
After they traded early breaks, Monfils took advantage of Federer's consecutive forehand errors to break in the ninth game and he served out the set on his first occasion.
Federer's charges to the net in a tight second set were loudly cheered by the fans but his lack of consistency from the baseline proved decisive. He failed to build on a 5-3 lead in the tiebreaker, losing the final four points.
“I never felt like things were really happening the way I wanted them to be going during the whole match,” Federer said. “The way I felt at 53 sort of showed how I really felt, I guess.”
It was Monfils's second consecutive win on clay over the 17-time Grand Slam champion after he beat him in straight sets during last year's Davis Cup final, and Federer's earliest loss in Monte Carlo since 2009.
“I'm surprised I was able to increase my power so quickly,” said Monfils, who recovered from a hip injury he picked up in Miami. “I was able to hit hard from the baseline, to be aggressive. I'm very pleased with the match.”
Against Isner, Nadal showed the grit and patience that have defined his 10-year reign on clay. The nine-time French Open champion waited until the middle of the third set to secure a break after withstanding an avalanche of aces and groundstrokes delivered by the 15th-seeded American.
“I lost in three sets in Rio. I lost in three sets in Indian Wells. I lost in three sets in Miami,” Nadal said. “So to break this streak of three-set losses was important.”
Nadal, who dropped to fifth in the rankings after his poor start to the season, fended off two set points in the tiebreaker but was broken in the ninth game of the second set when he went long with a forehand.
The turning point came in the fourth game of the decider, when Nadal earned two break points with a passing shot. He converted the first one with a thundering forehand winner and punched the air in his trademark celebration.
Nadal next faces clay specialist David Ferrer, who beat him at this stage of the tournament last year.
“He's one of the players who have been able to win more matches this year, so he is confident,” Nadal said. “He is one of the best players of the world. He is a good challenge for me. I'm excited to play that match tomorrow.”
The fifth-seeded Ferrer needed 2 1/2 hours to oust Gilles Simon, 6-2, 6-7 (3), 6-1.
Earlier, Dimitrov capitalized on Wawrinka's awful display to advance, as the seventh-seeded Swiss got off a sluggish start and never found the right balance.
Wawrinka was hoping to advance deeper in Monte Carlo, where he won his maiden Masters title last year. But he could not follow up his solid opening-round showing, hitting 13 unforced errors as Dimitrov raced to a 4-0 lead.
Widely regarded as a future star of the game, the 23-year-old Dimitrov broke for 3-1 in the second set after the former Australian Open champion shanked three forehands. He set up a meeting with Monfils when Wawrinka sailed yet another forehand out of court to drop his serve.