I think the big difference is where the sport stands in the heirarchy of the respective genders.
Tennis is THE #1 sport for women. As a result, they have had some of the greatest female athletes of all time (Navratilova, Graf). Conversely, there is no one that ever played men's tennis who I would remotely consider among the best male athletes of all time.
Why? A couple of reasons. Men have long had more options for a viable sports career. And there is the "macho" stigma. Tennis has always been labelled a "sissy" sport (and that's one of the more polite terms), thus discouraging those who wanted to prove how "manly" they are from chosing it. And contrary to what Johnny Mac wants to believe, that stigma persists today.
Look at the average male athlete. An average male athlete that stands 6 feet tall, and has a medium frame, is going to weigh about 185 pounds. In most sports, the physique is larger than that. In tennis, however, a build of 6'2" or 6'3" and 170-175 pounds is quite the norm. This puts them on the thin end of male athletes.
Tennis is a sport of technique. Proper mechanics are more beneficial than large muscle. Thus, small-framed men can excel at it. Now, if you add in societal factors, you get a situation where men's tennis becomes the province of the thin. Larger athletes will be steered toward more "macho" sports. And the country-club history of tennis as a genteel sport has lent itself to recruiting thinner athletes who don't want to associate with the "ruffians".
Being thinner gives no advantages in tennis, in terms of speed or endurance. It's just that those who have the speed and endurance of top tennis players, AND have medium or larger frames and the strength that goes with it, will gravitate to other sports.
And don't forget the economic factor. Decent players in other sports make far more than their tennis counterparts; often with the money in guaranteed contracts. Why play a sport where you have to continually be successful when you can earn more elsewhere, and not have to produce consistent success?
In women's sports, tennis is at the top of the totem pole. You earn more playing tennis than any other women's sport. Thus, many of the better athletes come to tennis. This not only includes athletes of every size, but also of every economic and ethnic background. Tennis gets a far bigger % of the cream of the women's athletic pool. Hence, many more athletes with muscles.