I figured out what the problem is and you guys are right. The light that I changed from spot to infinite has the light probe inadvertently attached to it.  I must have tried to use it as the IBL earlier. I've removed the light probe image from the infinite light but kept it on the IBL and now it looks close to what bagginsbill's got.