Many of them are re-elected because they are in "safe" districts just as Rangel and Pelosi are. You make a perfect point for term limits.
Then based on seniority rules and "safe" districts the current old guard is the only group that's going to make decisions because your philosophy keeps electing freshmen senators and representatives.