I think it depends on the "overall" look, the type of hat, and the person.
For example, the classic example of the baseball cap. It can be casual, and give off a "neutral" impression (based on your criteria). However, if it's worn backward, it can give off a number of different impressions: immaturity, "punk", or possibly even "douche".
A trilby, fedora, or top hat may give off a "classy" vibe, or it may not. Again, it depends on the whole package. I don't know if it's an "honesty" or a "closed/open" thing. Same thing with a cowboy hat. I mean, if someone is wearing "western" style clothing, a cowboy hat can look really good. But it can look ridiculous if someone is wearing shorts and sneakers, and thus the person looks ridiculous.
I mean, I think someone can give off an "immature but honest" vibe, if they have the right hat, and their general demeanor.
So I think it's a "total package" thing.