The first thing I now check, if I notice this issue, is whether there are any image_map nodes in my shaders that have texture filtering set to "none".

As far as I understand it, this prevents the image textures being cached in memory and means more (single threaded) disk IO...