Create your own robots.txt file

A robots.txt file is a short text file that resides in your home directory. Before search engines spider your site, they look into this file to see which files/file types and/or directories they are not allowed to see. This freeware utility will assist you in the creation of robots.txt files.
As a little ‘extra’ it also includes a block to many unwanted spiders that only crawl your site in order to collect the email addresses stored on your pages. (SPAM-bots)
Fill in the fields replacing the default values with your own ones. Create your robots.txt file, save it on your desktop and upload it at the root of your site.

Your website URL
Disallow all robots:
Your site won’t be cralwed by any robots including the safe ones like google.
[If checked -> click submit now]
Directories or single pages to exclude:
for enter /cgi-bin/
for enter /images/
for enter /free/tibet.html
for enter ../subdomain/private/
for enter /other/
for enter /other/
Disallow unsafe robots:
Include our sorted list of 135 additional unsafe robots that crawl your site for spamming or other dirty activity.

Default robot access

Additional rules



Files or Directories

Robots.txt File