How much memory does the GPU have? It's probably too small to handle much data - for Iray the whole scene has to load into the GPU's memory if it is to be used. You could test with a really simple scene, such as a sphere primitive sitting in splendid isolation.