A zero ambient value means no ambience at all.  However, IBL intensity alone set too high will washout shadow details.  Decrease it's intensity to a considerably low value as a remedy when using linear workflow.  Otherwise, the effect appears as you posted.