My own experience (lately I've been rendering scenes with many characters and things in them) is that so long as the thing is out of view (and this includes raytracing, so just moving it off to the side out of camera view won't work if you've got reflections that could see it) the render speed is reduced pretty much to the same as if you deleted it from the scene entirely.

I've got scenes where, if I don't need a character or object for that particular render, I move them outide the room, or drop them below the floor, or stuff them into a nearby wall or column to get them out of the way. Or simply make them invisible, either entirely, or to raytracing and/or the camera.