Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Robots.txt Generator

Robots.txt file is in the root folder of your website to help search engines to access the file. With the use of website crawlers, or robots it reviews your website. They may access your data you do not want them to, e.g. admin page. This can be done by using Robot.txt file. For this the best SEO tool is Robot.txt Generator availabel at websitechecker.ca

 A robots.txt file may have more than one rules. This helps you to block or even allow access for a path in that website. For this refer to Robot.txt generator.

For completely blocking the index of a particular page, use meta tag on every page. You can tell them to not index a page, by inserting either of the code bits in the HTML head of your document as follows,

<meta name="robots" content="noindex,nofollow">

Following are some guidelines to be followed while creating robots.txt files:

Use any text editor which will be able to create a .txt file. Avoid using a word processor as it often saves files in a registered format and add unexpected characters. The robots.txt Tester too will help you to create a proper format.

  1. It must name it as robots.txt
  2. Your site can have only one robots.txt file. For instance, http://www.abc.com/robots.txt
  3. You can have more files for each file.
  4. Syntax
    • This a text file.
    • It may consist of one or more directives.
    • Each group may consist of multiple rules or directives, one directive per line.
    • Rules are case-sensitive.

Robot.txt generator will easily generate the file for you.

 

 

 



PropellerAds