The `X Factor` boss - who has previously also judged `American Idol` and `Britain`s Got Talent` - accused the `Rocket Man` singer of being selfish and money-orientated after he accused the shows of being boring and exploiting the artists who take part.
He said: "Well, look. This is somebody who charges what, a million dollars a private gig? Two million dollars, you know? I don`t know whether he`s concerned about himself -- maybe he is."
Simon, 51, also insists he won`t take critics such as Elton seriously until they use their wealth and success to help aspiring musicians in the same way his shows do.
He told TV talk show host Piers Morgan: "They always bleat on that we`re not giving other people a chance. And I always want to say to them, `I tell you what, you just made a million dollars off your last private gig. Go and give it to a bunch of young musicians you care about, put them in the studio. Go and nurture them. Go and spend some time looking after them. Then I`ll buy your argument.`
"They`re only worried about themselves."
Last year, Elton revealed he turned down appearing on the judging panel of `American Idol` because he found reality television "a**e-paralysingly brain crippling".
Meanwhile, UK `X Factor` presenter Dermot O`Leary has been told he will not be fronting the programme when it launches in the US because Simon wants to adopt a dual-host format and is seeking a "non-conventional" female singer and a male "pin-up" musician.
Despite missing out, Dermot, 37, insists he is not too upset as he fared better than other British hopefuls.
He said: "I am obviously disappointed that I didn`t get the American gig. I would have loved a crack at it. At least I was the only Brit to make the shortlist."