Myanmar is "unable and unwilling" to investigate its abuses against Rohingya Muslims, a UN rights envoy has said, bolstering calls for the country's generals to be hauled before an international court.