By Agence France-Presse
The inferno raged through the eight-floor building of a circuit board maker, located in an industrial district in Taoyuan City.
Seven firefighters were trapped by large fallen objects as they tried to search for people inside the factory.
They were rescued early Sunday but only two survived. The two others killed were factory workers from Thailand.
Seven other firefighters were injured, five of them suffering burns from an unidentified liquid, according to the fire department.
Traces of diesel and toxic chemicals were also found at the scene, it said.
The disaster follows a string of recent accidents in Taiwanese factories, raising questions about safety standards.
A huge fire broke out at a refinery of the state-run CPC Corporation in January, also in Taoyuan. There were no injuries in that incident.
In December, six Vietnamese workers were killed in another Taoyuan factory blaze that erupted in the middle of the night.
The workers were found to have been staying in an illegal dormitory at the factory, highlighting the often sub-standard living conditions for migrant workers on the island.