Must Read: Google Robots.txt Parsing Weirdness
Google search bot (Googlebot) parses robots.txt file to find excluded sections of the site like any good webbots. However unlike the other bots, Google bot behaves differently when you have a section for all bots (*) as well as a section specifically for Googlebot.
Let's say you have a section in robots.txt for all bots beginning with:
User-Agent: *
Let's also assume that you do not have any section specifically targeted to Googlebot. In this case Google bot complies with the global directives (applicable to all bots). However now add a section specifically for Googlebot like this:
User-Agent: Googlebot
I assumed that it will comply with both the directives for all bots (*) as well as the directives specifically for Googlebot. As per robots.txt checker in Google Webmaster tools, in such case Googlebot only complies with the directives specifically targeted for Googlebot and ignores the global directives for all bots (*), even when the directives are non-overlapping.
Other bots including that of Google do not demonstrate this idiosyncrasy.
Filed under Company, Google, Headline News, SEO, Web |
|
RSS 2.0 |
Trackback this Article
|
Email this Article
You may also like to read |





































July 7th, 2008 at 11:44 am
For anyone interested, here is the user guide for robots.txt:
http://www.robotstxt.org/robotstxt.html
July 22nd, 2008 at 2:54 am
One more reference from Google Webmasters Tools
http://www.google.com/support/webmasters/bin/answer.py?answer=40360&query=robot&topic=&type=