I think one of the reasons Mat Smith works for me is he looks, well, slightly alien all by himself. Human, but not quite...if you know what I mean.  That makes the character a bit more realistic to me and I've been watching since the old Pertwee episodes.