Absolutely, I've known guys who were absolute chads, but completely oblivious of it, and women chased them with no luck. Then I've known super attractive guys who knew fully well, but they all came off as absolute douches, partied hard at the wrong time of their lives with all the women, and now fat, bald and not in a high income bracket. Interestingly, a lot of the ugly geeks are now financially secure and fit (in our forties) and are now seen as attractive.
By it's very definition, there are far more ugly guys than physically attractive guys. And a vast majority of those ugly guys are having sex. So it's simple stats why your argument doesn't hold good.
I'm older. Long married with two children. It's slowly dawning on men that attractiveness is poison when it comes to women. The supply-demand is so skewed, that the costs far outweigh the advantages. Men, find yourself a nice simply homely family oriented woman who can cook (but you should too!) and spare yourself a lifetime of misery.
And hyperfertile. I've heard of women that got pregnant from the sound of the guy unzipping as he was going to pee. Poor guy is on the hook for child support for the rest of his natural life.
While you're right, put yourself in the husband's place. Part with a little cash, get the ugly frigid old harpy out of the way ( by her own choice), and get hot fresh ripe succulent pussy. Seems like I deal I wouldn't pass up. Would you?