If you're using ray-tracing, then by all means, leave it on. But if you're not using ray-tracing, turn it off. To use ray-tracing, you have to have at least one light set to ray-traced shadows. If none of your lights are set to ray-trace, then there's no point in having it on.

It's the same with displacement. If you're using displacement, make sure you have "use displacement" checked. But if nothing in your scene is using the displacement node, uncheck it.

This is what I meant when I said "it depends on the image." Sometimes you'll be using ray-tracing, sometimes you won't. Sometimes you'll be using displacement, sometimes you won't. Some objects/textures require a higher shading rate than others. Etc. It depends on the effect you want.