The funnyman and actor shrugged off a $29 million (£21 million) windfall to return as optimistic Buddy the Elf because the film's premise was too similar to the original 2003 holiday comedy.
"I would have had to promote the movie from an honest place, which would've been, like, 'Oh, it's not good. I just couldn't turn down that much money,' " Ferrell told The Hollywood Reporter.
"I thought, 'Can I actually say those words? I don't think I can, so I guess I can't do the movie!' "
Will admitted there was a lot of pressure on him to land a successful film when "Elf" was released after leaving comedy sketch show "Saturday Night Live" following seven seasons.
And his agent's texts about the early test screenings didn't help.
"He was like, 'Well, the family one went great, but we could really get eviscerated in this next one. I'm looking at a bunch of what look like USC (University of Southern California) frat boys about to go in,' " Ferrell recalled.
"Then later I hear, 'No, that group actually liked it, too.' "