In this case I would say absolutely, because like it or not, the Confederate flag has become a divisive and inflammatory symbol to the vast majority of Americans.
When so many prominent and not so prominent hate groups have adapted what once was regarded as a symbol of pride, well, to quote The Princess Bride: "I do not think it means what you think it means."
The swastika for thousands of years was used to represent life, sun, power, strength, and good luck. But in our post-Nazi Germany world it is IMO the most repellant symbol known to man.
I would not rate the Confederate flag as being quite so repellant -- but there are some that would. The Confederacy was not the same as Nazi Germany, but there are people who have used the Confederate flag as a symbol of their own evil, and while those people did not have the power that Hitler did, some of them had just as much evil in them. The KKK comes to mind.
Displaying the Confederate flag may not make you a racist, but it undoubtedly symbolizes racism to many. And it HAS been used as a prop or cover by some. Consider this story, from Infoplease.com:
In 1956, Georgia adopted a new state flag that, like Mississippi's, incorporated the Confederate battle flag in its design. The flag, introduced two years after Brown v. Board of Education (1954), represented the Georgia legislature's protest against forced integration. Although some claim the new flag was adopted in anticipation of the Confederate Centennial in the 1960s, this argument was largely dismissed as disingenuous. In fact, the very sponsor of 1956 flag, former Georgia House floor speaker Denmark Groover, openly admitted forty-five years later that defiance of segregation was the motivating force behind the new flag, not historical sentiment.
And in Feb. 2001, in Tallahassee:
"Renovations required the removal of the four flags flying above the capitol. The flags, each representing a part of Florida's history, included Spanish, French, and British flags as well as the Confederate "stainless banner," the second national flag of the Confederacy. At the request of Governor Jeb Bush, none of the flags was again raised above the capitol. A spokesperson for Governor Bush commented that, "the governor believes that most Floridians would agree that the symbols of Florida's past should not be displayed in a manner that may divide Floridians today."