Since posting this I have turned off the Optix Prime acceleration and I have not noticed the issue again. Not exactly sure what Optix was buying me but my renders are much faster when they consistently use the GPU.