Home > On-site Optimization > What Is robots.txt ?

What Is robots.txt ?


When a search engine crawler comes to your site, it will seek a special file on your website. This file is called robots.txt tells the search engine spider web pages should be your site indexed and Web pages should be ignored. The robots.txt file is a text file (not HTML), which must be placed in the root directory

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s