A little history lesson:
Many on the side of taking the flag down have made much of the Stephens Cornerstone address and have used it for their assertion that the Civil War was entirely about slavery. However, this speech should be viewed in the context of the time it was delivered. The fact is that most politicians, North and South held many of the same views as Stephens. And the North was not trying to abolish slavery. The Corwin Amendment was passed by Congress at the same time and basically prohibited the Federal Government from interfering or impeding slavery. This was an attempt to get the Southern states to remain in the Union. This amendment was passed and sent to the states for ratification. Technically it is still pending. Both President Buchanan and President Lincoln endorsed it. Ohio, Maryland and Illinois ratified it. All the southern states had to do was ratify the amendment and slavery would have been protected from Federal purview. The Southern states chose to proceed with secession because they had other issues with the North such as the Morrill Tariff.
History is often much more complex than the sound bites people listen to.