"A lot of concrete progress has been made, but some differences remain," said Catherine Ashton, chief negotiator for the group of Britain, China, France, Russia, the United States and Germany.
The issue of stopping construction of a plutonium-producing plant was one of the key sticking points, according to French Foreign Minister Laurent Fabius.
A new round of talks to clinch a temporary deal on halting parts of Iran's nuclear programme in return for suspension of some sanctions was scheduled for November 20-21 in Geneva.
"Our goal continues to be a comprehensive solution to prevent a nuclear-armed Iran," German Foreign Minister Guido Westerwelle said.
The planned agreement fell through after intensive negotiations led by foreign ministers and other senior officials from all involved countries.
The talks included lengthy meetings between two leaders whose countries officially have no diplomatic relations: Iranian Foreign Minister Mohammad Javad Zarif and US Secretary of State John Kerry.