Hamilton was leading with 15 laps to go when his engine exploded at the Sepang International Circuit.
"No, no," cried the Englishman as flames appeared at the back of his car, leaving Ricciardo to take the lead ahead of teammate Max Verstappen.
Championship leader Nico Rosberg survived a first-bend spin involving Sebastian Vettel and a 10-second penalty to take third place and extend the gap over Mercedes teammate Hamilton to 23 points with just five races remaining.
Hamilton was almost in tears with his head in his hands after the dramatic end to his race which has dealt a severe blow to his chances of securing a third drivers' championship in a row.
He had looked all set to snatch back the lead from Rosberg, who was relegated from second on the grid to 21st after he was hit by Vettel on the first bend.
Vettel was forced to retire with a broken front suspension in the incident that also involved Verstappen.
Hamilton had got away cleanly from pole position and sped off into the distance, clear of the trouble behind him.
He looked to be heading for a comfortable win until his engine let go in the most spectacular fashion.
An ecstatic Ricciardo celebrated his first win of the season on the podium with a "shoey" -- drinking champagne from his driving boot.
"It was a race of challenges," said Ricciardo.
"Lewis got the lead and had his problems. It went the other way in Monaco, and I will take this today," said the Australian, referring to his botched final pit stop which handed Hamilton victory in Monaco earlier in the season.