Optimizing your Robots.txt for search visibility – @araghuwanshi6

Today, we’ll highlight some technical questions to avoid that can really hurt your search visibility and rankings

Website owners, use the file/robots.txt to instruct about robot web site? This is called the robots exclusion protocol. You can use a special HTML tag to tell robots not to index the content of a page, or/and scan for links to follow.

• Noindex:  The meta noindex tag prevents the page from being included in the index.
  

  Improve Your Search Engine Visibility

<META NAME=”robots” CONTENT=”noindex”>

• Nosnippet: Meta tag description “nosnippet” prevents the appearance of the page in the search results below, and also prevents the caching page. Although this tag won’t keep your pages from the index, it can have a significant negative impact on how your pages appear in the results.

<META NAME=”GOOGLEBOT” CONTENT=”NOSNIPPET”>

• Nofollow: The purpose of the “nofollow” tag, not to be confused with the rel = “nofollow” link attribute prevents search engine bots for any link on a page.

<META NAME=”robots” CONTENT=”nofollow”>

• Non-descriptive Title Tags: Title tags of your pages are probably the way more important that your web pages communicate what they are about to search engines. If your tags title doesn’t tell the search engines what your comments pages, the search engines will be less likely to understand what your page talking about and be used for a relevant search query.
• Non-semantic URL structure: URL of the Web site are another way that communicates your website from search engines. Your URLs do not pass semantic information about what pages are about, if the search engines will not take these signals. While this issue may not be serious, like “noindex”, in a competitive search legal, these signals can be the difference that separates you from your competitors in the eyes of search engines.

5th October 2009, Matt Cutts explains why a page that is disallowed in robots.txt may still appear in Google’s search results.

 

2nd July 2009, Matt Cutts explains If a page is disallowed in the robots.txt, will a link to this page transfer/leak link juice?”

 

Tuesday, March 16, 2010, Matt Cutts explains if there was at least one unfollowed link in a webpage, all links were considered unfollow? transfer PageRank?

See the Robots Exclusion standard: http://www.robotstxt.org/wc/exclusion.html

Alok Raghuwanshi is an India-based SEO Professional, and author of an SEO and social media marketing blog. Take a look and let me know what you think by including your comment below. Thank you and have a great day.

Follow him on

 Facebook Profile  @araghuwanshi6  LinkedIn Profile  YouTube Channel  Google