**Paloth :
What you see is Ambient occlusion beeing way to harsh.
Same goes for Chip room. Ambient Occlusion and/or self shadowing is not very pronounced in the real world as you say, it is however another light model, one that interacts with direct and indirect illumination/shadows very well if calibrated.
One could benefit rendering out the Ambient occlusion as a seperate pass and do an aditive blend post.
The umbra ( shadow place thats completly blocked from an opauge object like this ball ) is just natural. you can see the penumbra ( soft edge of shadow ) is more profound on the last image chip posted.
The Antrumbra ( area that is hit directly from source) seems to be smaller on the first image but its not, as the source hasnt changed in size, only intencity.

You will find that Chips image contains 2 very normal shaodw phenomenas as they would apear in our own world, Ambient occlusion and the interaction between mesh and a direct light.

The ambient occlusion is still present in the 2nd image, this is the shadow you see under the ball which reach out under neath it. that will always be there.

note : you can easily check if a shadow is having the correct amount of falloff in the penumbra ( the area between full shadow and no shadow ) by knowing th fact that for each 100cm the sun travels the penumbra increases 1cm ( beeing our own sun's dimentions )

Sun's distance from earth (center, but the few kilometers to the surface aren't going to matter much as we'll be taking averages elsewhere destroying superaccuracy anyway): 1.49610^11m
Sun's diameter: 1.39210^9
Disc angle: atan (1.39210^9 / 1.49610^11) = 0.533°

Project back to the distance between our object casting a shadow, and the surface receiving it:
penumbra = 1m * (1.39210^9 / 1.49610^11) = 1m * 9.3048110^-3 = 9.3048110^-3 ~= 0.00930481m, or 9.30481mm****

you can see a small example i made here with both light models in action.
Note the big penumbra sweeping over the floor and then note the legs on the chairs at the dinner table.

**