The "Blood Diamond" star shares 10-year-old Kenzo with the former model-turned-fashion designer, but claims he was barred from even talking to his boy on Sunday, June 16.
"It would have been nice to see my son for Father's Day," he told a photographer while out in Los Angeles on Monday.
"It would have also been nice, if I couldn't see him, to at least talk to him, right? Like all fathers, right?"
Hounsou claimed it had been so long since he was allowed to spend quality time with Kenzo, he couldn't remember when they last saw one another.
"I can't even recall myself, how about that?" said the 55-year-old, who had previously taken his son with him to the Hollywood premiere of "Aquaman" in December (18).
Despite his bitterness towards the unexpected separation from his son, Hounsou insists he's not about to take the battle to court.
Asked if he was going to take legal action to remedy the situation, he replied, "Listen, bro, it's no custody battle..."
Although Hounsou wasn't granted the opportunity to spend the holiday with his boy, he still marked the occasion on social media, sharing an older photo of himself and Kenzo hanging out together.
He captioned the shot, "#happyFathersDay to all fathers."
Hounsou and Lee Simmons dated for five years until their 2012 split.
The beauty also has three other children - teenage daughters Ming and Aoki with her ex-husband Russell Simmons, and a four-year-old son named Wolfe with her current husband, former banker Tim Leissner.