The players were spotted in a notorious red light district of Jakarta in their national jerseys last week, JOC officials told a press conference, saying the quartet had been ordered to leave immediately.
News that Yuya Nagayoshi, Takuya Hashimoto, Takuma Sato, Keita Imamura had been booted out will come as a major embarrassment for Japan, who were forced to send a swimmer home from the last Asian Games in 2014 for stealing a journalist's camera.
"I just feel a sense of shame," Japan's chef de mission Yasuhiro Yamashita told reporters.
"We deeply apologize and intend to give the athletes thorough guidance from now on."
The basketball players had dinner after leaving the Games village and are believed to have been solicited by touts on the street to go to a hotel with women, Yamashita added.
"I would like to humbly apologize to the Japanese public, the JOC and everyone who supports basketball for this deplorable incident," Japan basketball chief Yuko Mitsuya said in a statement.
"We will decide on the appropriate punishment for the four players once we have heard all the facts. We need to work harder to make sure this kind of scandal does not happen again."
Japanese swimmer Naoya Tomita was expelled from the Asian Games in Incheon, South Korea, four years ago after being caught on video stealing a journalist's camera.
It's also far from being the first case of sexual misconduct at a major multisports event, which typically draw thousands of athletes, officials and fans from around the world.
At the 2014 Asian Games, an Iranian official was kicked out for the verbal sexual harassment of a female volunteer, and a Palestinian footballer was accused of groping a female worker at the athletes' village.
In April, at the Commonwealth Games in Gold Coast, Australia, a Mauritian official was accused of sexually assaulting a female athlete during a photo shoot.
Organisers say about 18,000 athletes and officials are visiting Jakarta and co-host city Palembang for the Asian Games, a regional Olympics featuring 40 sports.