Humans are already borderline genetic failures because lack of genetic diversity caused by wars, lords hoarding all women, mass rapes, etc. Evolution is not fast enough to deal with those issues. Many health issues humans have today are already result of lack of genetic diversity caused by ice ages.

If only a minority of men reproduce, we will find in 2 or 3 generations a lot of cousins dating each other in small populations. This will create a lot of genetic bottlenecks, and if people have even fewer children, it will be even worse because fewer people to date.

Some may say that that will not be a problem because next generation will be genetically superior thanks to hypergamy, but this is not accurate either because hypergamy will still pick for 20% top of those men until genetic diversity is even lower.

This happened in the past, it can happen in the future.

In the past, hypergamy was compensated by some enforced monogamy, but such mechanism would not be considered legal or even humanitarian today. Sadly, women cannot avoid being attracted to top 20% men, even if all men improve, only a few of them will be successful. Genetic banks also take samples of top men, not healthy genetic men that may not be top but are good enough to keep genetic diversity.

Bottlenecks that reduced genetic diversity were common throughout human history | Berkeley News