The stampede occurred during a stone-throwing ritual in the holy town of Mina near Mecca, the Saudi Civil Defence agency said.
At least 719 pilgrims were wounded.
The stampede occurred at an intersection on one of the two routes leading to a multilevel structure known as the Jamarat, where the pilgrims cast pebbles in a symbolic stoning of the devil, the Civil Defence said in a statement.
About 220 ambulances and 4,000 rescue workers rushed to the scene, which was sealed off. Pilgrims were redirected to alternative roads, the government-run agency said.
TV footage showed the bodies of pilgrims, clad in white seamless pieces of cloth, being carried on stretchers to ambulances.
Saudi authorities did not identify the nationalities of the victims Nearly 2 million pilgrims participated in the symbolic stoning of the devil, a major ritual in the five-day annual haj, which started Tuesday.
The incident came three weeks after a construction crane collapsed in Mecca's Grand Mosque, Islam's holiest site, killing 108 people.
The desert valley of Mina, 10 kilometres east of Mecca, has seen deadly stampedes in recent years, prompting the oil-rich kingdom to spend lavishly to boost safety standards during the stone-throwing ritual.
In 1990, 1,426 pilgrims died in a stampede inside a Mina tunnel after the ventilation system broke down. In 2006, 364 pilgrims were crushed to death during the stone-throwing ritual.
In the wake of the 2006 tragedy, the Saudi authorities built additional floors to the Jamarat building to ease crowding and barred pilgrims from moving in opposition directions in the site.
Muslims are expected to perform the pilgrimage, one of Islam's five pillars, at least once during their lifetimes if they have the financial and physical means to do so.