To further explain....
Lots of record (owned prior to passage of the comp plan) retain their original setbacks which in most cases are five feet.
Also, small or oddly shaped lots can be exempted from setbacks as well.
I will say regarding side setbacks that this is a very important number when talking about fires. When the Mussett Bayou fire broke out the homes saved were generally more than fifteen feet apart. The main reason for side setbacks is to allow emergency vehicles through.