I can't be bothered to in the Subaru. The airflow it gets is just... wrong. When I daily drove the Miata, though, the top was down unless it was raining. While I do live in Texas, I had many drives in sub-freezing temperatures with the top down. And these days, I daily a motorcycle, and do so regardless of weather.
So, yeah, I'm out there in it, whatever the weather may be. If you enjoy that, I'd suggest looking for a convertible or a motorcycle to have as "another" vehicle - the experience is vastly improved with the greater openness these vehicles offer.