I definitely don't think it delays speech. Since you say the word while signing the word, they pick up on both. Eventually they figure out that talking is much less "work" than signing. Also, kids are a lot less coordinated with their chubby little fingers, so there's a lot of things they will have a rough time signing. My signing child started talking at 9-10 mos (singing his abc's by 11mos), my non-signing child just started saying mommy at 22 mos!
eta: signing just gets kids through the times when their brains are ahead of the mouths, eventually the two catch up and things progress as usual. I also think of it as sort of being bilingual... kids who grow up in bilingual houses don't usually end up being stuck with one language, they learn both and understand more about language than children who only hear one language through their formative years.