A lack of direct flights between London and Baku means supporters face costs of over £1,000 ($1,300) just to make the 5,000-mile round trip before tickets and accommodation are taken into account.
Arsenal had initially complained over their small allocation, but confirmed to AFP that they have returned 2,200 tickets for the Gunners' first European final in 13 years.
AFP understands that Chelsea have sold just 2,000 of their 6,000 allocation.
UEFA denied reports that sponsors had also returned tickets and said they were confident that local fans will snap up the spare tickets to ensure there is a capacity crowd at the 68,000 Olympic Stadium.
"The majority of these tickets have already been sold to local fans in Azerbaijan and we are confident that the remaining ones will also be sold as demand is very high," UEFA said in a statement to AFP.
The Chelsea Supporters' Trust (CST) joined other supporters' groups in criticising UEFA for the location of the final, but also turned on their own club's handling of the affair.
"Baku is a totally unsuitable location for a major European final. The combination of cost, complexity in regard to travel arrangements and time off work has massively reduced the travelling support, including those who loyally and ordinarily go to all home, away and European matches," said the CST in a statement.
The CST also complained about the cost of charter flights put on by Chelsea and that the club has not extended the time for season ticket renewals in order to help fans' cash flow.
Arsenal midfielder Henrikh Mkhitaryan will also not travel to the final due to safety fears.
Mkhitaryan's native Armenia and Azerbaijan have feuded over Nagorny Karabakh since Armenian separatists seized the territory in a war that claimed about 30,000 lives in the early 1990s.