arcady opened this issue on Dec 24, 1999 ยท 6 posts
BillBay posted Fri, 24 December 1999 at 7:53 AM
A slightly more complex, but effective way to keep the spiders out of your images: (This is a quote from Altavista, but works for almost everyone of them.) "Excluding Pages with robots.txt Our crawler will find any URL connected to the main body of the Web through even one link. If you don't want your entire site to be indexed, we strongly advise that you take advantage of the Robots Exclusion Standard by setting up a robots.txt file. It only takes a minute, and gives you complete control over what fraction of your site is indexed. The file looks like: To exclude your site from all web crawlers use: User-agent: * # directed to all robots Disallow: /cgi-bin/sources To exclude your site AltaVista's web page index: User-agent: scooter # AltaVista web page search Disallow: /cafeteria/lunch_menus/ To exclude your site AltaVista's image, video and audio clip index: User-agent: vscooter # AltaVista Image Search Disallow: /personal/images/ Any URL matching one of these patterns will be ignored by robots visiting your site."