When we bought our house, it was going to be our primary residence. The north side of the bay ended up being closer to work, less vulnerable to hurricanes, and we just didn't want to have to deal with the fuss of getting on 98 every time we had to go somewhere.
When we first moved here, we rented about 100 yards back from the Gulf for a couple of years. It was enough to decide we wanted to buy away from crowds and storm worries, even though there were things I loved about it, and once in a while I do miss being able to just walk out the front door and cross the street to get to the Gulf. Plus, it was easier to find neighborhoods where 80%+ of the people lived here year round, and we wanted an area that seemed like more of an established community.
Owl's Head and the like would be actually be tempting to us in some ways, but we'd both have bad commutes from Freeport, and we'd take a big tax and interest rate hit compared to where we're at now.