Barcelona star Martens had earlier given the Dutch the lead in their last-16 tie at Roazhon Park with a brilliant backheel flick, but that contender for the most eye-catching goal of the tournament was cancelled out before the break as Yui Hasegawa equalised for Japan.
The Nadeshiko were the better team in the second half but were undone by yet another penalty in a World Cup dominated by spot-kicks, with Martens scoring after a Vivianne Miedema shot had struck the hand of captain Saki Kumagai from point-blank range.
The Netherlands, reigning European champions, go through to a quarter-final on Saturday against Italy, who defeated China 2-0 earlier in the day.
"It is history that we have made. We are really proud about that but we are not done yet here. We are really looking forward to the next game and we hope we can surprise more," said Martens.
She accepted the penalty award at the end had been harsh on Japan, adding: "It's not our decision, we are really happy with that and sometimes you need a bit of luck."
After winning the 2011 World Cup and losing in the final four years ago, Japan go home, ending Asian interest in France already.
Seven of the eight quarter-finalists are from Europe, with holders the United States the only exception. The manner of Japan's defeat here drew sympathy from the crowd, with loud jeers greeting the final whistle.
"It was a penalty, it struck my hand. Of course it is very hard to accept. I am sad but I know that that's football," admitted a tearful Kumagai.
The Netherlands had been the more impressive of these two teams in the group stage, backed by an army of travelling supporters decked out in orange.
They had far fewer fans backing them this time in Rennes, where the heatwave suffocating much of the rest of France has stayed away too.
Nevertheless, the Dutch began strongly and almost went in front inside the first five minutes as Miedema met a Martens cross from the left only to see her effort take a deflection off Aya Sameshima, hit the post and go behind.
- Touch of class -
Sarina Wiegman's side were rewarded for their enterprising start with the opening goal via a touch of class by Martens in the 17th minute, her near-post flick from a Sherida Spitse corner going through the legs of Yuika Sugasawa and in.
Japan's starting line-up included only four players who featured in the 2015 final defeat against the USA and Asako Takakura's youthful side were now in danger of following the tournament's other Asian representatives and going home.
However, their quick response to falling behind saw Sugasawa hit the post with a curling effort, and the Nadeshiko drew level two minutes before the interval.
The Netherlands were cut open as Mana Iwabuchi supplied Hasegawa, and she poked a first-time effort high into the net.
Japan built on that and were the better team after the break, with Dutch goalkeeper Sari van Veenendaal making excellent saves to deny Emi Nakajima and substitute Yuka Momiki.
In between, 11 minutes from time, Hina Sugita broke forward from midfield into the box and cut inside to hit a shot on her left foot that cracked off the crossbar.
Their opponents were hanging on for extra time until Miedema sent her shot off the arm of Kumagai, leaving the Honduran referee with little option but to give a penalty, and Martens made no mistake to send the Dutch through.