Try applying the background to the Alternate Diffuse instead of the normal diffuse channel and see if that makes your backgrounds any better. Set the normal diffuse channel to zero when you do it. The image shows what I mean, normal diffuse on the left, alternate diffuse on the right. The only side effect of the alt diffuse is that the background wont receive shadows.