The pair replaced Dermot O'Leary but were met with criticism.
A show source told The Sun: "Caroline’s a really experienced broadcaster but the combination of two presenters just didn’t work out this year.
“It’s unfortunate for her, but the decision gives her the opportunity to take on a major West End role, which is something she’s wanted for a long time.
A X Factor spokesperson said: "Details for the new series of The X Factor are still being discussed and will be announced in due course."