This may be related to the male variability hypothesis. Males tend to show more variance from the mean for traits with a normal distribution. So for attractiveness, you are more likely to find both particularly attractive and particularly unattractive men than women, since the latter are going to mostly fall closer to the baseline level.

But personally, I agree the difference between putting in effort or not can have a bigger difference for men.