The former couple embarked on a romance in 2001 after co-starring in action thriller "In God We Trust", later renamed "Hard Cash", and the actors began living together in New Mexico.
However, when the romance fell apart, so did Kilmer, reports People.com.
In his new memoir, "I'm Your Huckleberry", the 60 year old recalls, "Lord knows I've suffered heartache. But Daryl was by far the most painful of all."
"Neil Young, I always loved you, but I'm afraid I hate you now," he quips.
Kilmer also opens up about his longtime bond with Cher, his girlfriend in the early 1980s, crediting her for pulling him through some of the darkest days of his battle with throat cancer in 2015.
"Once Cher works her way inside your head and heart, she never leaves," he shares. "For her true friends, her steadfast love and loyalty never die."
He goes on to gush about Jolie, his co-star in 2004 movie "Alexander", noting she was "perhaps the most soulful and serious of them all" as he reflected on their off-screen dalliance.
"When people ask what she's like, I say she's like other women and other superstars, just MORE," Kilmer adds.
"I'm Your Huckleberry: A Memoir" is set to hit book retailers on April 21, 2020.