Sandra Bullock and Jesse James split in 2010 after the TV mechanic's infidelities hit the headlines, but she suggested she'd always stay close to little Summer, seven, after forging a bond with the child during the marriage.
However, after confessing he had no contact with Louis, the baby he and Bullock adopted just before the split, earlier this week, James has now revealed he and his ex never even speak.
During an interview with U.S. news show "Nightline", which will air in its entirety on Wednesday, James says, "I've never seen Louis since everything happened... (Sunny's) seen her a couple times, but that's it. But pretty much no contact for several months."
But James, who has since proposed to celebrity tattooist Kat Von D, is determined to the put the past behind him and move on with his life.
In a defiant statement on the show, he adds, "I can't worry about her (Bullock) anymore. I think I've spent a good chunk of the last five or six years worrying only about her and what she thinks and what I should do and, like, controlling all my movements and everything else. And I think it's time to worry about Jesse and making sure Jesse is happy."
James plans to wed his fiancee this summer.