Forum: Vue


Subject: The renderfarm experiment, part II (animation)

louguet opened this issue on Sep 16, 2005 ยท 16 posts


JavaJones posted Sun, 18 September 2005 at 6:20 PM

louget I'm not trying to knock what you've done in any way, I think you're providing some great info and I'm certainly appreciative. I'm only trying to offer advice from my own experience that might be helpful in ensuring accurate results. I have some personal experience in benchmarking and it's always been a subject of interest to me so I've done a good deal of research on it. One of the major consistent issues is minimizing non-related factors in your benchmark. For example if you're testing a game and want to find out what the maximum CPU bound framerate is, you want to ensure that the graphics card will not be a bottleneck, so you first use the fastest graphics card you have available, and 2nd you run the benchmark at extremely low resolution, since higher graphics resolutions depend almost entirely on the video card. The CPU will be stressed fully this way. The same principles apply to your situation of course. I'm sure I am not telling you anything you don't know, but perhaps this will explain to others what I am talking about. In any case it's all well and good to say that the overhead has no noticeable effect, but the real question is have you measured it? If so, then I'm sure you're correct and your results may reasonably used to extrapolate for other potential situations. But if not, and you are simply making a judgment call based on your experience, I'm sorry but I think that calls the results into some amount of question. The overhead is an unknown quantity - low, according to your experience, which I'm certainly willing to accept, but not necessarily immeasurably low, and when you're dealing with such short render times, as I said the influence of even a small overhead could be significant. Of course it may not be your intention that others extrapolate from your data to other potential situations, but people are likely to do so regardless. It is no duty of yours, but would certainly be appreciated - if you're going to publicize your results - if you would practice due diligence in ensuring their accuracy. It would be of benefit to all, including yourself. All that being said it's likely that if you retested with a longer rendering scene, the results would be similar and I might look a fool. But the principle of what I'm saying holds true. You simply can't safely make any assumptions when benchmarking, otherwise your results are largely useless. Last but not least I'd like to reiterate: please don't take offense to any of this. I am not saying I know better than you. I am largely regurgitating what I have learned in my own years of personal benchmark experience and especially learning from others who know more and have greater experience than I. Take that as you will, but above all please don't be upset by it. Life's too short. ;) - Oshyan