Modric, 33, was announced as the winner of the prestigious prize in a glitzy ceremony held in Paris. Ronaldo was second, with France and Atletico Madrid striker Antoine Griezmann completing the podium.
Paris Saint-Germain's teenage forward Kylian Mbappe came fourth, leaving Messi to settle for fifth position. Another Frenchman, Modric's Madrid teammate Raphael Varane, was seventh, just behind Liverpool's Egyptian forward Mohamed Salah.
"As a kid we all have dreams. My dream was to play for a big club and win important trophies," Modric said.
"The Ballon d'Or was more than just a dream for me and it is really an honour and a privilege to hold this trophy."
Mbappe, who does not turn 20 until later this month, picked up the consolation prize of the Kopa Trophy for the best young player.
The gala ceremony also saw a women's Ballon d'Or named for the first time, with Lyon's prolific Norwegian striker Ada Hegerberg taking that award.
Modric -- the oldest winner since Italy's Fabio Cannavaro, also 33, in 2006 -- was crowned after an outstanding year in which he starred in the Real side that retained the Champions League before heroically helping drag his country to the final of the World Cup for the first time in their history.
Croatia lost 4-2 to France in Moscow in the final in July.
The prize comes after Modric claimed the Golden Ball for the best player at the World Cup before picking up UEFA's prize for player of the season and FIFA's best player award in September.
Ronaldo and Messi were among 30 nominees for the Ballon d'Or, organised by France Football magazine and which takes votes from 180 journalists from around the world.
The Portuguese star played with Modric in the Madrid side that won the Champions League before moving to Juventus in a typically prolific year, while Messi has again been outstanding with Barcelona.
The pair had won the prize five times each in the last decade but both are now in their 30s and neither stamped their authority on the World Cup.
Messi finishes outside the top three in the voting for the first time since 2006.
The World Cup was always likely to play a key role, and France had hoped to see one of their stars take the award on home soil.
- 'Huge step' for women's game -
The last time a Frenchman won the Ballon d'Or came in 1998, when Zinedine Zidane was rewarded for starring as Les Bleus won that year's World Cup.
But perhaps the lack of a single standout French candidate cleared the path for Modric -- Hugo Lloris, Paul Pogba and N'Golo Kante had also been in the running, as had Tottenham Hotspur and England striker Harry Kane, the World Cup's top scorer.
"I won the Europa League and the World Cup, so it's been a great year, but it is true that it was a little bit disappointing after I was told what the podium was," said Griezmann.
Modric has won 14 trophies since joining Real in 2012, including four Champions Leagues. He could add to that as Real take part in the Club World Cup later this month.
A refugee at one point growing up during Croatia's War of Independence in the 1990s, his popularity in Croatia was hit by his testimony last year during the multi-million-euro corruption trial of former Dinamo chief Zdravko Mamic.
Prosecutors charged Modric for giving false evidence, but the charges have now been dropped.
Hegerberg, 23, won the women's prize after starring for French side Lyon as they successfully defended the Champions League title and won a 12th consecutive French title.
"I want to say thanks to my teammates because this would not have been possible without them, my coach or our president Jean-Michel Aulas," Hegerberg said.
"I also want to thank France Football. This is a huge step for women's football."