If you change the gamma correction one way or the other, you need to change at least the lighting as well. If you have a scene that's acceptable to you without GC and turn on GC, of course it's going to look different. The first thing is that with proper GC, you don't need as many light, or lights as bright.

Since you say you're using diffuse IBL, of course everything is going to be washed out. The first thing you'd have to do is significantly decrease the strength of the IBL. Like, a lot. Before GC I'd use a diffuse IBL in the 15-20% range to get something that looked right (albeit to me). After GC was introduced, maybe 5-10%. And odds are you have too many lights (if they are all focused on the same thing).