By Agence France-Presse
The Geneva-based body decided to "dispatch urgently an independent international fact-finding mission" to Myanmar, particularly focused in northern Rahkine state, in a resolution adopted by consensus.
The mission must provide an oral update in September and a full report by this time next year, said the resolution backed by the European Union.
The decision marks a partial diplomatic victory for Myanmar's new civilian government led by Nobel peace laureate Aung San Suu Kyi.
The United Nations special rapporteur on rights in Myanmar, Yanghee Lee, had urged the council to set up a Commission of Inquiry -- the world body's highest level probe -- to document anti-Rohingya violations.
But Lee's call found little support, including among EU powers with major influence in the council.
Earlier this month, Lee told reporters that European leaders wanted to give Suu Kyi's nascent government more time before launching a top-level inquiry, which may have implicated top military officials in crimes against humanity.
Myanmar has been staunchly opposed to a Commission of Inquiry. The country's rights council ambassador, Htin Lynn, also called the move to send a lower level investigation "unacceptable".
Myanmar has launched its own domestic probe into possible crimes in Rahkine and appointed former UN chief Kofi Annan to head a commission tasked with healing long-simmering divisions between Buddhists and Muslims.
Japan, which supported Friday's resolution, said in a statement that the UN should wait until the national probe led by Vice-President U Myint Swe reports and only dispatch international investigators "if necessary".
Rohingya who have fled to Bangladesh have told the UN rights office that soldiers executed babies in front of their mothers in Rakhine state as part of campaign to terrorise the Muslim minority.
Myanmar has long faced international condemnation for its treatment of the Rohingya, who many in the Buddhist majority reject as illegal immigrants from Bangladesh.
The issue has reached boiling point in recent months after the army launched a bloody crackdown in the north of Rakhine after deadly attacks on several police border posts in October.