The National Front won 133 parliamentary seats, seven fewer than previously, while the opposition People's Alliance won 89 seats, the final tally by the elections commission showed.
The stock market surged by as much as 6.8 per cent Monday, after the polls were conducted peacefully, following a campaign marred by clashes.
Bursa Malaysia's benchmark FBM KLCI index rose by up to 123.76points to 1,818.53 in early trading, but tapered down to 1,745.31 by midday, 50.54 points up from opening.
Prime Minister Najib Razak hailed the election result, and called for "national reconciliation" after campaigns marred by politicalviolence.
"We are looking forward to reject political and racial extremism, and work towards a more moderate and accommodating environment." But a source within the National Front said Najib could be facing challenges within the party after failing to win more decisively.
"Even before the elections there were already rumblings within the UMNO about Najib being too liberal," the source told dpa.
The UMNO, or United Malays National Organization, is the leading party within the National Front alliance.
"He has to consolidate his base within UMNO immediately, otherwise he might go the way of Pak Lah," the source added, referring to former prime minister Abdullah Badawi.
Badawi was ousted in 2009 and replaced by Najib after the 2008elections, which saw the ruling coalition's number of seats drop below the two-thirds majority for the first time, from 198 to 140.
Najib blamed Sunday's poor result on negative "racial" sentiments, especially from the ethnic Chinese, who make up a third of thepopulation.
"I expected it but I did not expect it to this extent," he said.
Several prominent coalition members were unseated, including the chief ministers of the state of Malacca, Rustam Ali, and of Johor, Adbul Ghani Othman, as well as Federal Territories Minister Raja NongChik Zainal Abidin and Transport Minister Kong Cho Ha.
Opposition leader Anwar Ibrahim accused the government of cheating his People's Alliance of victory, and vowed to contest the results.
"We want the elections commission to explain the irregularities" and "resolve the disputed seats" he told reporters.
A survey a week earlier put the opposition 1 percentage point in the lead. Ibrahim had said a high turnout would work in their favour.
Turnout in the predominantly Muslim South-East Asian country reached 80 per cent in Sunday's election, the highest in its electoral history, Deputy Elections Commissioner Omar Ahmad said.
Malaysia has been governed by the National Front since it gained independence in 1957.