# This is a standard robots.txt file that will allow ALL web spiders # to crawl your web site. # Visit http://www.searchengineworld.com/robots/robots_tutorial.htm # for more information on robot.txt files User-agent: * # Allow all spiders in to start Disallow: ## Stop spiders accessing particular pages Disallow: /tools Disallow: /scripts Disallow: /images Disallow: /members Disallow: /js Disallow: /css Disallow: /photos ## Stop Google getting images. User-agent: Googlebot-Image Disallow: / ## Stop the Vast bot (vast.com) User-Agent: Vast_Bot Disallow: / ## Stop the Turn It In bot (turnitin.com) - I highly doubt students will be plagerising findacar! User-agent: TurnitinBot Disallow: /