As a large framed 6'2" male I got a lot of negative reactions when I was very fit with short hair. Back in collage I ended up growing very long hair to move from the thug stereotype to Fabio. Which was kind of amusing, but also a good way to move things in a positive direction. Except that one cab driver who hated people with long hair because the hippy movement killed off his local barber shop.

So, yes people will judge you. But, you can also distract people by giving them something else to focus on which is under your control. While I am not suggesting you walk around in clown shoes, but if you did that's going to be the first thing on everyone's lips.