The former `Two and a Half Men` star is currently involved in a custody battle with estranged wife Brooke Mueller over their 23-month-old sons Bob and Max and the actor - who also has daughters Sam, seven, and Lola, five, with former spouse Denise Richards, as well as 26-year-old Cassandra from a previous relationship - insists he has no plans to expand his family further.
Charlie - who is currently living with two women, porn star Bree Olsen, and part-time model Natalie Kenly - told gossip website PopEater: "I can`t seem to keep the ones I already have. So no, I don`t want any more kids."
Bob and Max were removed from their father`s custody last week after Brooke - who has previously been treated for drug addiction - became concerned about Charlie`s erratic behaviour.
The 45-year-old actor is now fighting her move, which also included taking out a temporary restraining order against him, and friends insist he wouldn`t have taken action if he wasn-t concerned about Brooke`s parenting abilities.
One insider said: "You can say whatever you want about Charlie, but he is a good father.
"If Charlie wasn`t so concerned with Brooke`s mothering skills and her addictions, he would be more than happy to let her have full custody, just as Denise does with his daughters."
The insiders claim Charlie would happily share custody with Brooke and see the boys every other week if he thought she was up to caring for them, but he doesn-t believe they would be safe with her.
The pal added: "He can`t stand by and allow them to live with someone he doesn`t think is a good parent."