One other aspect of the extra rays is when using premium effects such as soft shadows and blurry relections or DOF. The effects of lower ray numbers (or oversample rate) are a serious graininess in what should be a smooth visual effect. 36 (or 6x6 oversample) may work depending on the individual image but 64 (8x8) is definitely better for premium effects as a rule of thumb.

Pass no temptation lightly by, for one never knows when it may pass again!