The star played the primary antagonist, an artificial intelligence programme tasked with defeating Keanu Reeves' hero Neo, in the Wachowskis' 1999 science-fiction classic and its two sequels.
However, the Aussie star revealed he won't be starring in a fourth movie, due out next May (21), as shooting dates clash with him appearing on the London stage alongside Lesley Manville in "The Visit", which runs at the U.K.'s National Theatre from 31 January to 13 May.
Asked in an interview with Time Out magazine whether he would be returning, Hugo said, "I'm not, no," before going on to explain the reasons for his absence.
"It's unfortunate but actually I had this offer (for 'The Visit') and then the offer came from 'The Matrix', so I knew it was happening but I didn't have dates," he explained. "I thought (I) could do both and it took eight weeks to work out that the dates would work - I held off on accepting (a role in 'The Visit' during that time)."
"I was in touch with Lana Wachowski, but in the end she decided that the dates weren't going to work. So we'd sorted the dates and then she sort of changed her mind. They're pushing on ahead without me."
The 59-year-old also revealed why he did not return to play Red Skull in the final two "The Avengers" movies, and was replaced by Ross Marquand, explaining that he refused to participate after Marvel chiefs reneged on an agreement to increase his pay.
"The promise when we first signed the contracts was that the money would grow each time," he added. "They said: 'It's just a voice job, it's not a big deal'. I actually found negotiating with them through my agent impossible. And I didn't really wanna do it that much. But I would have done it."