Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

Robots.txt Generator

Robots.txt file is in the root folder of your website to help search engines to access the file. With the use of website crawlers, or robots it reviews your website. They may access your data you do not want them to, e.g. admin page. This can be done by using Robot.txt file. For this the best SEO tool is Robot.txt Generator availabel at websitechecker.ca

A robots.txt file may have more than one rules. This helps you to block or even allow access for a path in that website. For this refer to Robot.txt generator.

For completely blocking the index of a particular page, use meta tag on every page. You can tell them to not index a page, by inserting either of the code bits in the HTML head of your document as follows,

<meta name="robots" content="noindex,nofollow">

Following are some guidelines to be followed while creating robots.txt files:

Use any text editor which will be able to create a .txt file. Avoid using a word processor as it often saves files in a registered format and add unexpected characters. The robots.txt Tester too will help you to create a proper format.

5. It must name it as robots.txt
5. Your site can have only one robots.txt file. For instance, http://www.abc.com/robots.txt
5. You can have more files for each file.
5. Syntax
   • This a text file.
   • It may consist of one or more directives.
   • Each group may consist of multiple rules or directives, one directive per line.
   • Rules are case-sensitive.

Robot.txt generator will easily generate the file for you.