One of them, the former director of the infamous Hoa Lo prison, recalls verbally sparring with the famous inmate and says McCain's refusal to budge on his views eventually earned his admiration.
"It was his stubbornness, his strong stance that I loved when arguing with him," retired colonel Tran Trong Duyet told AFP.
In the decades following the Vietnam War, McCain -- who died Saturday at the age of 81 -- forgave the enemies who once held him captive, and helped reconcile the two countries that today enjoy strong ties.
His five and a half years in prison began in October 1967 when McCain was thrown into the French-built jail after his Skyhawk divebomber was shot down over Hanoi's Truc Bach lake.
Fished out with a broken leg and two broken arms he was shipped to the cold, crowded facility where some 500 prisoners of war were held.
His captors quickly learned McCain's father was a navy admiral, and the young prisoner soon developed the nickname "Crown Prince".
The early years were grim.
McCain was held in solitary confinement and suffered from dysentery. For months on end, he was fed only bread and pumpkin soup. He communicated with fellow inmates by tapping codes on the thick concrete walls.
In his memoirs, McCain wrote that solitary "put me in a pretty surly mood" and that he would ward off depression by hollering insults at guards.
And then there were the interrogations and beatings.
"Ropes were put on me and I sat that night bound with ropes," McCain wrote after his 1973 release, recalling one brutal session.
"For the next four days, I was beaten every two to three hours by different guards. My left arm was broken again and my ribs were cracked."
-- English classes, jokes --
Duyet denies McCain or others were mistreated and says he punished any fellow guards who stepped out of line.
"There was no torture, Vietnamese people saved him," Duyet said in an interview earlier this year at his home in the port city of Haiphong, where he displays both photos of American POWs and more recent images of himself in military uniform posing with US officials.
By the end of his long years in prison, Duyet said his relationship McCain started to warm.
"Out of working hours, we considered each other friends," he said. "He taught me English... he had good teaching skills."
In his post-prison writings, McCain said things got easier for him in the early 1970s, which he called the "coasting period".
He read propaganda texts about Vietnam communist revolutionary leader Ho Chi Minh, and was allowed to roam the prison yard with fellow Americans, who named different sections of the compound after Vegas hotels.
His former jailor preferred to focus on the rosier memories, recalling how they joked, shared stories about family and travel, and even dished about women.
"We laughed together and agreed that women are the same everywhere -- they like flattery, they like to sulk, and they're jealous," he told AFP, smiling.
- 'America's POW' -
McCain left the navy in 1981 for a long career in politics, most notably as Republican Senator for Arizona. But his "Hanoi Hilton" experience resonated throughout his life.
Donald Trump said during the campaign for the presidency that McCain was "not a war hero" because he was captured in Vietnam.
The controversial comments inflamed his throngs of supporters, many of whom saw his time as a POW as a defining experience.
"John McCain will always be America's prisoner of war, that experience is now inseparable from the name John McCain and the person that has been John McCain," said Alvin Townley, author of "Defiant" about American POWs held in Hoa Lo.
McCain was also celebrated for his role in reconciling the United States and Vietnam, which in the half century since the war's end have become close allies.
"(His) openness towards Vietnam and the willingness to revisit not only the country, but his experiences there have certainly helped heal a lot of wounds," Townley said.
McCain visited the Southeast Asian country several times after the restoration of diplomatic ties in 1995, even returning to the "Hanoi Hilton" -- now a popular tourist attraction -- for an emotional meeting with another former jailor.
Duyet never got a chance to reconnect with McCain, but imagined what he might say if he had.
"If he came to Vietnam, I would greet him, not as a former prisoner and a jailer, but as two veterans, from both sides of the battlefield, now meeting again in the spirit of reconciliation," he said.
When informed of McCain's death, Duyet said Sunday he was "so sad."
"Please if you can, convey my condolences to his family for me."