{"id":1072,"date":"2010-09-13T10:26:40","date_gmt":"2010-09-13T07:26:40","guid":{"rendered":"http:\/\/www.webseoanalytics.com\/blog\/?p=1072"},"modified":"2010-09-13T10:26:40","modified_gmt":"2010-09-13T07:26:40","slug":"robotstxt-metarobots-relnofollow-the-impact-on-seo","status":"publish","type":"post","link":"http:\/\/www.webseoanalytics.com\/blog\/robotstxt-metarobots-relnofollow-the-impact-on-seo\/","title":{"rendered":"The robots.txt, META-robots & rel=nofollow and their impact on SEO"},"content":{"rendered":"

\"metarobots-seo\"One the most common mistakes that many Webmasters and SEOs make is that they misuse<\/strong> the robots.txt, the META-robots and the rel=nofollow anchor attribute. All the above affect the indexing<\/strong>, the crawling<\/strong> and the PageRank distribution<\/strong> and as a result they can heavily affect the SEO status of a website and its rankings<\/strong>. In this article we will see what is the impact of robots.txt, META-robots and rel=nofollow on SEO and how you should use them to improve the rankings of your website.<\/p>\n

Robots.txt<\/h2>\n

The robots.txt (aka Robot Exclusion Standard) is a special text file that is placed in the root folder of a website (for example: www.example.com\/robots.txt). It is used by Webmasters in order to give instructions about their websites to web spiders and crawlers. This file contains information about which pages should not be crawled by the web robots, what the maximum requested crawl rate should be, where the XML sitemap is located etc. One of the common uses of robots.txt is to prevent<\/strong> search engines from crawling<\/strong> a particular page or folder.<\/p>\n

Below we\u2019ll see the answers of few common questions about robots.txt that will help you prevent major SEO mistakes<\/strong>:<\/p>\n

I blocked a page\/folder\/website by using robots.txt. Will Google\/Bing\/Yahoo crawl that page?<\/h4>\n

No they will not crawl it.<\/strong> All major Search Engines respect the Robot Exclusion Standard. This of course will not protect you from the spam bots since they do not respect the above standard.<\/p>\n

I used robots.txt to block a page\/folder\/website. Will it appear in the search results?<\/h4>\n

YES!<\/strong> Blocking a particular page or a part of your website by using robots.txt does not mean that it will not appear in the Search Engine Results. The page will not be crawled<\/strong> by search engines but if they find enough information about the page\/site on DMOZ<\/strong> or in the links<\/strong> of other pages, they will show it in the search results.<\/p>\n

\"alexa-robots.txt-blocked\"<\/div>\n

The title and the data found in the SERP will be based on the information that the search engines collected from other websites.<\/p>\n

<\/embed><\/object><\/div>\n

I blocked an internal page with robots.txt, will it receive PageRank?<\/h4>\n

YES! <\/strong>PageRank algorithm does not take into account whether a page is blocked by robots.txt, so the incoming links that point to those pages do pass link juice. Additionally since those blocked pages can`t be crawled by the search engines, their outbound links will not be found and thus they will be considered as dangling pages<\/strong>. This means that those pages evaporate<\/strong> all the PageRank that they receive so this leads to PageRank losses<\/strong>.<\/p>\n

<\/embed><\/object><\/div>\n

META-robots<\/h2>\n

The META-robots is a METAtag that affects the search engine crawling and the way that PageRank flows from a webpage. This META-tag specifies whether a particular page should be indexed<\/strong> and whether the outgoing links should be followed<\/strong>. The default value of this tag is “index,follow” which means that this page is allowed to be indexed and all the links should be followed (except the ones that have rel=”nofollow” in them). The valid values of this META-tag are: index<\/strong>, noindex<\/strong>, follow<\/strong>, nofollow<\/strong>, none<\/strong> (none means noindex and nofollow [thank you Sebastian for the reminder]) and all<\/strong> (all means index and follow).<\/p>\n

Below we examine the answers of common questions about META-robots:<\/p>\n

I blocked a page by using the noindex META-robots value. Will Google\/Bing\/Yahoo crawl that page?<\/h4>\n

Yes they will crawl it.<\/strong> The search engines will visit that page and extract all the links of the page.<\/p>\n

So if they crawl the page, it will also appear in the search results. Correct?<\/h4>\n

Absolutely NO!<\/strong> You should not confuse the terms crawling and indexing. It is one thing to get your pages crawled by a search engine and a completely different thing to get it indexed. Anyway, once the search engines crawl the webpage, they will identify the meta-robots noindex value and they will not index<\/strong> it. This means that this page will never appear<\/strong> in the search engine results.<\/p>\n

I am a bit confused with the follow and nofollow values? How do they affect the PageRank distribution?<\/h4>\n

The follow<\/strong> value means that all the links found in this page flow PageRank (except the ones that contain the rel=”nofollow” attribute in them). The nofollow<\/strong> value means that all the links of the page are nofollowed and thus no PageRank flows from this page. This means that the page which blocks all of its links by using the meta-robots nofollow value, is considered a dangling page<\/strong> and it evaporates all of its PageRank.<\/p>\n

I blocked an internal page with META-robots noindex, will it receive PageRank?<\/h4>\n

YES! <\/strong>As we mentioned earlier, the PageRank algorithm does not take into account whether a page is blocked by robots.txt or meta-robots, so the incoming links that point to those pages do pass link juice. The main difference with the robots.txt is that the pages that use the META-robots noindex<\/strong> value, can flow their PageRank to other pages if the follow<\/strong> value is also used. This means that if their links are followed the PageRank of the page will not be evaporated.<\/p>\n

rel=”nofollow”<\/h2>\n

The rel=”nofollow” attribute is used in the anchor links that we want to block so that they do not pass<\/strong> any link juice to the target pages.<\/p>\n

Below we examine the answers of common questions about rel=nofollow:<\/p>\n

I blocked all the links of my site that point to a particular page by using the rel=nofollow. Will Google\/Bing\/Yahoo crawl and index that page?<\/h4>\n

Nofollowing all the links of your website that point to a particular page (internal or external), does not mean that the search engines will not be able to index the target page. If they find a dofollow link pointing to that page, they will crawl and index it. So rel=nofollow should not be used<\/strong> as a way to prevent<\/strong> the indexing of a page.<\/p>\n

<\/embed><\/object><\/div>\n

Do search engines crawl the links that are blocked with rel=nofollow?<\/h4>\n

Theoretically<\/strong> search engines do not<\/strong> crawl<\/strong> the target page of nofollowed links until they find at least one followed link that points to that page. Practically<\/strong> in many cases the major search engines do crawl nofollowed <\/strong>links. Additionally have in mind that in some cases those links can help you speedup the indexing of a new page. Of course the fact that in some cases they do crawl those links, does not mean that they pass any link juice\/PageRank to them (see below).<\/p>\n

How PageRank distribution is affected by nofollowed links?<\/h4>\n

First of all Nofollowed links do not pass<\/strong> any PageRank <\/strong>to the target page. Period.<\/p>\n

In the past (before 2009), during PageRank calculations, the nofollowed links were just like they did not exist and they did not affect positively or negatively the PageRank distribution. Unfortunately after 2009, an update took place and the nofollowed links started to evaporate the amount of PageRank that they receive. In order to learn more about this update and in order to find out how to solve this issue read the article “The PageRank sculpting techniques and the nofollow issue<\/a>“.<\/p>\n

Summary<\/h2>\n

Note that you should be very careful when using robots.txt and META-robots since you can heavily affect your website`s indexing and your Search Engine Rankings. From SEO perspective, the best way to prevent a page from appearing in the search results is to use the noindex,follow<\/strong> META-robots value. In this way not only the page will not appear in the SERPs but also most of the PageRank of the page will pass to the other pages of the website through its links.<\/p>\n

The table below summarizes how the major search engines handle robots.txt, META-robots and rel=nofollow.<\/p>\n

\n\n\n\n\n\n\n\n
<\/td>\n\n

Crawl<\/strong> <\/strong><\/p>\n<\/td>\n

\n

Appears in SERP<\/strong><\/p>\n<\/td>\n

\n

PageRank Evaporation<\/strong><\/p>\n<\/td>\n<\/tr>\n

robots.txt<\/strong><\/td>\n\n

No<\/p>\n<\/td>\n

Yes, if it receives enough links.<\/td>\nYes, all of the PageRank of the page is evaporated.<\/td>\n<\/tr>\n
META-robots noindex<\/strong><\/td>\n\n

Yes<\/p>\n<\/td>\n

No<\/td>\nNo, if the links of the page are followed.<\/td>\n<\/tr>\n
META-robots nofollow<\/strong><\/td>\nYes, if other followed links are found pointing to the target page.<\/td>\nYes, if the target page receives links from other webpages.<\/td>\nYes, all of the PageRank of the page is evaporated.<\/td>\n<\/tr>\n
rel=”nofollow”<\/strong><\/td>\nYes, if other followed links are found pointing to the target page.<\/td>\nYes, if the target page receives links from other webpages.<\/td>\nYes, some amount of PageRank of the page is evaporated.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Last but not least, if you use extensively the rel=nofollow attribute in your website, you can avoid PageRank losses<\/strong> by using the method described in the article “PageRank Sculpting: You can still use rel=nofollow<\/a>“.<\/p>\n","protected":false},"excerpt":{"rendered":"

One the most common mistakes that many Webmasters and SEOs make is that they misuse the robots.txt, the META-robots and the rel=nofollow anchor attribute. All the above affect the indexing, the crawling and the PageRank distribution and as a result … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":1073,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[6,10],"tags":[],"aioseo_notices":[],"_links":{"self":[{"href":"http:\/\/www.webseoanalytics.com\/blog\/wp-json\/wp\/v2\/posts\/1072"}],"collection":[{"href":"http:\/\/www.webseoanalytics.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.webseoanalytics.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.webseoanalytics.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/www.webseoanalytics.com\/blog\/wp-json\/wp\/v2\/comments?post=1072"}],"version-history":[{"count":0,"href":"http:\/\/www.webseoanalytics.com\/blog\/wp-json\/wp\/v2\/posts\/1072\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/www.webseoanalytics.com\/blog\/wp-json\/wp\/v2\/media\/1073"}],"wp:attachment":[{"href":"http:\/\/www.webseoanalytics.com\/blog\/wp-json\/wp\/v2\/media?parent=1072"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.webseoanalytics.com\/blog\/wp-json\/wp\/v2\/categories?post=1072"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.webseoanalytics.com\/blog\/wp-json\/wp\/v2\/tags?post=1072"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}