When optimizing your web site most webmasters don’t consider using the robot.txt file. This is a very important file for your site. It let the spiders and crawlers know what they can and can not index. This is helpful in keeping them out of folders that you do not want index like the admin or stats folder.

Here is a list of variables that you can include in a robot.txt file and there meaning:

1. User-agent: In this field you can specify a specific robot to describe access policy for or a “*” for all robots more explained in example.
2. Disallow: In the field you specify the files and folders not to include in the crawl.
3. The # is to represent comments

User-agent:  *
Disallow:  

User-agent:  *
Disallow:  /cgi-bin/

User-agent:  googlebot
Disallow:  

User-agent:  *
Disallow:  /admin.php
Disallow:  /cgi-bin/
Disallow:  /admin/
Disallow:  /stats/