By Agence France-Presse
Abu Sayyaf leader, Abraham Hamid, had led the kidnapping of several foreigners from a tourist resort in the volatile southern Philippines last year, two of whom were later beheaded.
"The death of Hamid is a big blow to the (Abu Sayyaf) as it neutralised one of the notorious bandits and will degrade their capability for spotting and kidnapping victims in the future," said regional military spokesman Major Filemon Tan.
Two other militants were killed alongside Hamid in the shootout with Malaysian police in Lahad Datu in eastern Sabah, he said.
The Abu Sayyaf beheaded two Canadian hostages after demands for millions of dollars were not met, but released two others, a Norwegian and Filipina, after ransoms were believed to be paid.
Tan said Hamid had also been involved in the kidnapping of four Indonesian crewmen in April.
There have been a spate of kidnappings of Malaysian and Indonesian sailors at sea in recent months that have been blamed on the Abu Sayyaf.
While Hamid and two militants were killed, Sabah security forces have arrested two others, Tan added.
Sabah police chief Abdul Rashid Harun told AFP the incident was the Malaysian authorities' first direct confrontation with suspected kidnappers in the waters off eastern Sabah.
On his blog, Malaysian Prime Minister Najib Razak praised his security forces and said Kuala Lumpur and Manila would cooperate to fight the recurring kidnappings.
The Abu Sayyaf, a loose network of militants based on remote islands in the southern Philippines, has defied more than a decade of military offensives.
The group was formed in the 1990s with seed money from Osama bin Laden's Al-Qaeda network, but has been on a lucrative kidnapping spree in recent years.