The One Direction heartthrob has reportedly left the presenter - who, at 32, is 15 years older than the singer - after ending their three-month romance so he can concentrate on breaking America with his band.
A source told The Sun newspaper: "It was never going to work and Harry knew that.
"The split happened very quickly after Harry made the decision to call it a day.
"He spoke to Caroline and explained that he thought it was best they break up. They`ve been spending a lot of time together so it`s a blow to Caroline, who was serious about the relationship.
"After their romance became public it put a lot of pressure on young Harry.
"He`ll be spending a lot of time in the US with the band this year so knows it will be hard to keep things on track."
Despite the split, the pair remain on good terms but Harry knows it would be difficult to continue their union when he will be away so much.
Another source told the Daily Mirror newspaper: "Harry is being realistic about the whole thing.
"He`s going off to America in a couple of days and he wants to be able to commit fully to making the band a success in the States.
"They`ll be back for the Brit Awards next month, but all in all they`ll be away for two months. It`s a long time to be apart especially with so much going on. Harry`s young. He`s just not ready for a long-distance relationship.
"He really likes Caroline and he doesn`t want to hurt her. But he doesn`t want to be tied down.
"Caroline confided in pals about the split, confirming it`s over. She is upset, but they are still on good terms."