Federer, playing only his second match of a 2019 comeback to clay after three years away, needed two hours to defeat French showman Monfils 6-0, 4-6, 7-6 (7/3).
Osaka fell for the second time in two months to Federer's Swiss compatriot Belinda Bencic, exiting 3-6, 6-2, 7-5. Bencic also beat the double Grand Slam winner in March at Indian Wells.
Men's top seed Novak Djokovic, however, motored into the last eight as he hammered Jeremy Chardy 6-1, 7-6 (7/2) while five-time champion Rafael Nadal dispatched Frances Tiafoe 6-3, 6-4.
Nadal will next play Stan Wawrinka after the Swiss completed a national hat-trick on the day, beating Japan's sixth seed Kei Nishikori 6-3, 7-6 (7/3).
Defending champion Alexander Zverev came from breaks down in both of the last two sets to get past error-prone Pole Hubert Hurkacz 3-6, 6-4, 6-4 and now plays eighth seed Stefanos Tsitsipas, a 6-3, 6-4 winner against Fernando Verdasco.
Federer admitted he struggled to get past the talented Monfils, seeded 15th, before recording his 1,200th career win.
It took a comeback from 4-1 down in the final set after sweeping the first set in 18 minutes, combined with a dominating tiebreak for the Swiss fourth seed to advance.
Federer said nothing was decided until the end of a match played in difficult swirling winds and challenging conditions.
"I had a tough run of five losing games late in the second and into the third," said the 20-time Grand Slam champion who will face Austrian fifth seed and recent Barcelona champion Dominic Thiem for a place in the semi-finals.
"I just hung in there, saving two match points (to force a final-set tiebreaker).
"It was a very stressful situation. Now that I'm through it, I'm very excited with the win.
"This match gave me a lot of information, it showed how tricky clay can be with the wind and everything. But I tried to take the bad bounces and the rest and just play positive tennis."
- 'Too negative' -
Against an inspired Bencic, Osaka served for the match in the third set leading 5-4, but could not touch a backhand return winner.
"When you play against a good player, you have to expect that they are going to play great at times," Osaka said.
"I feel like I was negative on myself, but I tried to fix it in the third set. But my service percentage was down."
Djokovic, who now boasts a staggering 66-2 record against French players, said he is slowly lifting his clay game.
"I felt good on the court in general and played well," Djokovic said.
"Coming into the match I had confidence knowing I'd never lost to my opponent."
The Madrid champion in 2011 and 2016 is searching for his top clay form as he begins the chase for a fourth straight Grand Slam title when Roland Garros starts in a little over two weeks.
Djokovic next faces ninth seed Marin Cilic after the Croatian put out Laslo Djere 4-6, 6-3, 6-2, winning 11 of the last 13 games.
Thiem, the 2018 runner-up, defeated Monte Carlo winner Fabio Fognini 6-4, 7-5.
Two-time women's champion Simona Halep overcame Ashleigh Barty 7-5, 7-5 to reach her fourth Madrid semi-final.
Romania's third seed and former world number one will next play Bencic.
"For me, it's a huge victory, it's a huge match I played today," said Halep.
Former US Open winner Sloane Stephens downed Petra Martic of Croatia 6-4, 6-3 in her quarter-final and next faces either defending champion Petra Kvitova or Kiki Bertens for a place in the semi-final.