Boundaries matter, but the issue is that the phrase goes hand-in-hand with "Well, if that's all you're good for....." It's very dangerous to encourage either mindset. That can lead to people doing things and using that belief as motivation.

If I need an example: sexual objectification of women.

If a man only cares about tits, it won’t matter if she’s covered up, “respectable,” young, old, shy, loud, religious, married, liberal, conservative. He’ll only think in cup sizes. He’s already made up his mind women are only good for a nice pair of tits to look at.

Also, either that mindset doesn’t account for abuses of power or it’s condoned. Part of the reason the MeToo movement became a thing is because there were  menusing job opportunities as blackmail for sex. Predators/abusers love taking advantage of vulnerable people, so to imply that abusive men are allowed to act a certain way is victim blaming.