The actor is currently appearing in the 80s-set ABC comedy but was almost dropped from the cast after people complained about his language between takes.
"I was saying some really stupid, silly things that I can't believe that anyone would find offensive, but to each his own," he told the New York Post. "Every time I stand up (on the set) I go, 'Oh, my (female genitalia).' Which is silly and stupid, I clearly don't have (female genitalia)."
The star continued: "I said to them, 'I understand if I worked at an insurance office and there's a bunch of desks - that's inappropriate. But in a comedic atmosphere it's stupid. They said, 'What if we fire you?' and I said, 'I'd love to have it announced Jeff Garlin got fired for saying (that)'."
The star is enjoying a particularly successful career run, with his stand-up special, "Our Man in Chicago", streaming on Netflix, and season 10 of "Curb Your Enthusiasm" dropping in January (2020).
Now, he explains he's taking a break from further live comedy shows, telling the publication: "I was filming 'The Goldbergs' at the same time that I was filming Curb Your Enthusiasm, and at night I would go out and work on my stand-up set. Right now I'm taking a little break from stand-up just because I need to re-energise myself."
Season seven of "The Goldbergs", also starring Wendi McLendon-Covey and Sean Giambrone, is airing now.