Property line issue

We have a neighbor who has lived next door to us for about two years now. Just himself and his wife. Probably in their 60's. Before they bought the house this man that was living in the house and remodeling it at the same time removed the property line stake that was in front of the telephone pole between our property and that property. It was the property stake the city put there upon surveying the property. When we moved in this house back in 1994 this elderly lady lived in that house and had lived in it probably 30 years or more. She took us up to the telephone pole and uncovered the property stake buried in the ground right in the middle, showing it to us. Then covered it back up. First, he was just coming over around two or three feet into our yard, we figured it was because of him having to use our yard to turn his mower around. This morning upon mowing his yard he came 6 or 7 feet over into our yard mowing that section of our yard all the way down to our fence line at the back yard as if it is his yard. He mowed our entire side yard right by our bedroom window including all the way up to the street, a good 7 foot wide area. My husband says he doesn't care if he mows it as long as he is not trying to claim that part of our yard. I feel he is. I just went up there and put a pressed board stick that I spray painted red on the top part down in the ground right in front of that telephone pole to mark the property line. Should we say anything or let it go? We don't want to have to pay for another survey and have another stake put there, however I feel that stake does need to be replaced. We have a copy of the survey of the property when we bought the house showing where it starts. Just wondering what other people think. Thanks.