I'm happy to think with you, but I do lack the info to do so.
?1 do you use IDL lighting. The IDL pass can be slow as well, and telling about issues. I've the impression you don't and use direct lights only.
?2 is Task manager telling you anything (cpu use, ram use, nr of entries, ...) and are you rendering as a separate process
?3 render subpasses withing firefly are:
 = loading textures (takes time when you've got a lot and texture cache hasn't caught them yet). Lots of disk activity but not that much ram usage.
 = geometry handling (takes time when you've got loads of subdivision, displacement map, etc). Now your ram will go up.
 = rendering on a per-block basis. Hotspots will reveal as rendering goes on but publishing the block-results in the view window stops
So: where does it stuck

Knows performance killers:
 * materials with raytraced reflections while not being opaque (transparency >0.0 somewhere), especially in combi with IDL. Eyes, glasses, stones in jewelry. Refraction alone is like opaque.
 * max bucket sizes <16 (or >128)
 * displacement bounds set (much) larger than the displacements in your scene
 * pixel samples >4 (3 = display quality, 4= fine print). Required for DoF and motion blur only. Personally, I render a ZDepth pass (in Aux render data) and apply DoF in post. Much faster, much better result. The Firefly DoF noise keeps being grainy at any setting.

So, please help us to help you. 

- - - - - 

Usually I'm wrong. But to be effective and efficient, I don't need to be correct or accurate.

visit www.aRtBeeWeb.nl (works) or Missing Manuals (tutorials & reviews) - both need an update though