The attached shows what the effect of the shadow map can have. The right hand side of the image has the map size set to 20, the left it is at the default of 256. As you can see there are lines across the image where the map size is 20. Do they look like the streaks you mentioned?

If they don't then the cause is probably either AO in the IBL. The answer will be to change the bias setting in the scene AO options