I don't quite understand the insinuation that's often made that it's somehow a bad reflection of someone's moral character if they put value on physical appearance in their partners or potential partners. This is generally what the implication of calling someone 'superficial' or 'shallow' is in the context of dating.

The way I see it though just like someone may get enjoyment out of someone's company because they make them laugh or have deep conversations about life with them, they can get enjoyment out of simply looking at their face or body and presumably doing other things with them as well. What is the difference here that makes the first examples acceptable but the latter requires a pejorative to describe it?

I mean I could make an evolutionary argument for why appearance is valued and especially by men, but I guess you also could for something like rape so that may be dismissed as a 'naturalistic fallacy'.

With rape the harm done is obvious however, what is the harm that's being done by valuing physical appearance?

I can see it maybe being unwise, in that physical features tend to be more transient than psychological ones, meaning that they should have less weight when it comes to judging someone as a candidate for a long-term partner, but it could also be seen as unwise to eat too much junk food and we don't tend to call someone a bad person if they indulge in it. When it comes to smoking cigarettes, it's second hand smoke which people tend to have the moral issue with.

Is it because of the notion that physical appearance is more permanently fixed than other factors, making valuing it in a partner somehow less 'fair'? First off it's not obvious that this is the case as psychological traits tend to be highly genetically heritable too, and secondly wouldn't this for instance make sexual orientations other than bisexual are immoral too? To avoid controversy let's say it's 100 years ago before sex change operations were available.