Terminusnord uploaded this item to try to find the amount of bandwidth wasted by "bots" alias "spiders". I support him in this.

Useless sphere - do not download (Poser) For Non-Commercial Use Only Stand Alone Item Notes & Requirements By: terminusnord Aug 19, 2002 PC/Mac 18 downloads - - - - - -

The download count is at 9 am 20 Aug 2002 by UK time. - How much does Renderosity check who is downloading what to check if any of its users are spidering the freestuff? - As the freestuff is hosted on other sites, how much of this spidering is not via Renderosity but straight to the users' sites? The various web-search sites (Alta Vista, Googol, etc) must get their information from somewhere, about what text words occur in what web pages, and that would be spidering all of the internet that they can get at. Ditto people collecting email addresses for spamming.

Is it really spiders that download stuff? How can they? I mean, you have to click the "download" button, it's not like it's a text link... However I did notice the number og downloads of the "useless sphere" but I kinda reckoned it was someone who couldn't controll their curiosity G I know I almost couldn't resist the temptation to download it just to see if it really WAS a useless sphere L But if bots/spiders really ARE downloading stuff from this place, then how can it be stopped? And how can you check for it?

And, is a simple sphere really completely useless? :-)

1.) Most spiders/crawlers download a file called robot.txt first. In this file you can specify files which should not be included in the analyze. I get around 30 hits per month for this file (404 error, because this file dont exist).

2.) I guess renderosity uses a pearl/cgi script to initiallize the download. Im not sure whether spiders can access and interpret cgi files.

I think Renderotica had trouble with spiders some while back, and instituted some sort of spider-bashing device, but my memory is hazy on the details.

I'm convinced there's spiders DLing from rosity. For the past couple weeks browsing the galleries has been laggy, which it never has been before. I remember a year ago Poserworld and Propsguild were nearly brought to a crawl because of automatic DLing programs. I think thats why a lot of sites started using javascript popups for freestuff (or became paid sites).

A simple javascript password would stop indexing. And stop spiders and bots in there tracks. If seen a few other sites put a password script on the entrance page along with the password for just this reason.