Quote - But it wouldn't just be a case of finding the distance to the camera for each vertex?

It's more to do with how efficient the floating point math can be done.
Checking if a point is in a simple bounding box is a few compares, calculating the distance requires a square root calculation (which you wouldn't actually do, you'd keep everything squared).
Even checking against a frustrum (which requires a matrix transform) is comparatively quick.