The seven-storey building in the beach town of Sihanoukville, owned by a Chinese company, collapsed with workers inside.
"We have pulled out a body and we see two more bodies still stuck between debris," Yun Min, the governor of Preah Sihanouk province, told AFP, adding 13 people were injured.
Images showed excavators digging into the flattened debris in the hunt for dozens of other workers feared to be trapped inside.
"There are more than 30 people in the pile of debris from the building," Information Minister Khieu Kanharith said in a Facebook post, citing witnesses.
He said all of the dead were Cambodian, including two workers and a translator.
Governor Yun Min said around 50 workers would normally have been on the site at the time.
"We have rescued 20 so far... we still can't say exactly how many are trapped in the debris," he said, adding some may not have been at work at the time of the accident.
Rescue workers in hard hats heaved chunks of broken cement from the flattened building, looking for anyone buried underneath, while the diggers shunted twisted metal to the side.
Medical workers attended to a shirtless injured man, as concerned crowds built up around the site.
The building belonged to a Chinese national who rented the land from a Cambodian owner.
Sihanoukville was once a sleepy fishing community before being claimed first by Western backpackers, and then wealthy Russians.
Chinese investment has flooded in in recent years, spurring a construction boom in a resort town known for its casinos which pull in mainland tourists.
There are around 50 Chinese-owned casinos and dozens of hotel complexes under construction.
Between 2016 and 2018, $1 billion was invested by Chinese government and private businesses in the Preah Sihanouk province, according to official statistics.
Cambodia, one of Southeast Asia's poorest countries, has notoriously lax safety laws and labour protections. Accidents are common at its building sites.