Keep on doin' what you're doin'. You are the authority in your life, and at Willow Manor and you have a good safety net of authority that you accept in your own life. It is enough. It's hard to deal with some nay-sayers but just shine them on and tell them your prayer is that they never have to go through such a thing.
The flat cap will take some getting used to but I don't really think it softens your look. It's just...different. I used to have one for a while but it just didn't feel right. I still have but the Professor wears it these days.