Prayut said the media speculations on Cabinet reshuffle caused him to lose concentration on his works.
He added that he would not replace those who can carry out his orders but those who could not perform would have to be replaced.
He said a reshuffle would take time because officials would have to first check qualifications of new ministers.
Prayut added that he would not change Deputy Prime Minister Prawit Wongsuwan, who is in charge of the security affairs.