Yeah, I think it's simply what a well-dressed figure in a position of authority would wear in the outdoors at that time. The riding boots were probably the gentleman's only alternative to town shoes - he wouldn't be seen dead in rough work boots. Once you're wearing riding boots, the jodhpurs are the most comfortable way to go, and khaki is a particularly good colour for spotting ticks on your clothing.

European plantation owners in the colonies would all have been wearing much the same getup, with a riding crop for pointing at minions. Maybe a famous French director started the beret thing.

Phil, while you're on a roll, did robbers ever really wear striped teeshirts and little black Zorro masks?