It happened many times that I intended to up-vote someone but downvoted him due to some human error. Is there any way to remove wrongly given reputation and fix it?

Unfortunately no. It's been discussed before but apparently is "one of those things" that's difficult to change in the software.

You can find other posts by the user and upvote them to counterbalance the accidental vote, but that only works if they *have* sufficient other posts... and those parts are also useful (please don't go an upvote random posts that aren't necessarily good just to counteract an accidental downvote).