Voting for or against someone, solely based on them being the incumbent, is absurd. Who are they running against? Like any election, you have to vote for the person you think is the best candidate.

Now, it could be a tie-breaker. I can't remember this ever happening, but say there are two candidates who are essentially equal. Their only real differences are on issues you don't care about. Then, sure, toss out the bum and put the new person in. Would you rather have someone who did some things you disagree with, or someone with whom you will disagree on everything?