Robots.txt File Generator

This generator creates a robots.txt file. The robots.txt file tells search engine spiders (robots) which files and directories they are NOT allowed to index. This helps to prevent incomplete site indexing as well as prevent exposing the files and directories that you don't want listed.