The group boats (aka head boats, party boats) typically carry 50 or more people. They allow kids but do not cater to them. They will usually discourage really young kids because the boats are dangerous (they pitch, the decks are wet and slimy, there are knives, gaffs and hooks all over the place, large fish jump around and can be dangerous when brought on board, etc.). If the child gets seasick, bored or sunburned they will not go back, which can make the kid pretty miserable (as well as his/her parents and other nearby adults). So consider yourself forewarned.
The one I have been out on most is the Swoop. I have heard good things about the New Florida Girl and Destin Princess. I think (am not positive) that the Destin Princess is the largest of the bunch which will help with seasickness, boredom and sun/heat. I do not have direct phone numbers for them, but they can be found by doing an internet search using phrases like "destin fishing boats", "Destin Princess", etc.
On edit: Also, be advised that the trip out is about an hour and a half, maybe two. Same amount of time coming back. I think your 11 and 14 yr. olds will be fine, but I question bringing the 4 yr. old.