Googlebot
Googlebot is the web crawler which Google uses to find web pages on the Internet.
Googlebot is known by it’s referrer tag, which looks like this:
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
The Google AdSense Googlebot
The Google AdSense bot is confusingly named and therefore causes frequent confusion as webmasters mistake it for Googlebot.
The Google AdSense bot crawls web pages which contain Google AdSense advertisements, in order to determine what advertisements are most relevant for that page.
The Google AdSense bots referrer tag looks like this:
Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)
Being visited by the Google AdSense bot will not get you into Google’s search engine index.
Quoting The Google AdSense FAQ:
Adding the Google AdSense ad code or WebSearch code to your site will not queue your pages for crawling by our main index bots. While our bot (Mediapartners-Google/2.1+) does crawl content pages for the purpose of targeting ads, this crawl is not associated with our main index crawl.
The Google Automatic URL Removal Console Bot
If you have used the Google Automatic URL Removal System to remove a web page from the Google index, you will receive a visit from the UrlConsole Googlebot.
The Google UrlConsole bots referrer tag looks like this:
googlebot-urlconsole
The Googlebot FAQ
For more information on Googlebot, read the Googlebot FAQ.
|
|
- Internet Bots
An Internet Bot is a software application that does repetitive and automated tasks in the Internet that would otherwise take humans a long time to do. The most common Internet bots are the spider bots which are used for web server analyses and file data gathering. Bots are also used to provide the required higher [...]...
- How Do I Optimize my Website for Search Engines?
SEO, an abbreviation for Search Engine Optimization, is the practice of optimizing a website for web search engines. With a comprehensive optimization technique implemented on a website, the search engines will be able to index and crawl it in an effective way. The goal of website optimization is to get your website listed in the [...]...