Home     Blog

robots.txt

robots.txt is a text file which can be used to restrict web robots to accessing your web site only in ways of which you approve.

This robots.txt file blocks Google's Imagebot from the entire web site:

User-agent: Googlebot-Image
Disallow: /

For more information on robots.txt, read A Standard for Robot Exclusion.

Check the Syntax of your robots.txt

Several web-based tools are available which will retrieve the robots.txt file from your web site and check it for syntax errors.robots txt robots.txt

Bad Robots

Some web robots will use up considerable amounts of bandwidth and system resources, while returning little or no practical benefit to the web site owner.

For your convenience, we maintain a list of some of those bad robots in robots.txt format.

VN:F [1.9.17_1161]
Rating: 0.0/10 (0 votes cast)
Follow Daniel Memetic on

Comments (2)

 

  1. xolas says:

    new dios “kurko”(EXTENDERLO)

    VA:F [1.9.17_1161]
    Rating: 0.0/5 (0 votes cast)
    Reply
  2. aaa says:

    asdasd sasd asd

    VA:F [1.9.17_1161]
    Rating: 0.0/5 (0 votes cast)
    Reply

Leave a Reply

Related Posts

  • List of Bad Web Robots

    This list of bad web robots was originally compiled by funender. Some of these robots are designed to copy entire web sites, which can utilize a great deal of bandwidth. Others harvest e-mail addresses for spammers. Most benefit their owners, while providing absolutely no benefit to the web site owner. You must decide for yourself [...]...


  • How to Find Broken Links

    Broken links annoy the visitors to your web site, which lowers you percentage of return visitors. In addition, broken links on your web site may negatively affect your search engine rankings. Finding broken links on your web site can be accomplished by either a downloadable program such as Xenu Link Sleuth, or by using a [...]...


  • How to Prevent Downloading of Your Entire Website

    Preventing Web Site Downloading Using robots.txt The first step is to disallow the downloading programs in your robots.txt file. To do this, you will need to define which bad robots you wish to disallow. Disallowing bad programs in robots.txt does not prevent all web site downloading, because many bad programs simply ignore the contents of [...]...


  • Should I Use WWW in my site name?

    Some users type “www.topbits.com” into their web browsers when they want to reach this web page. Other users type in only “topbits.com”, leaving off the “www” portion. On a technical level, the two names do not refer to the same domain object. Google, and other search engines, often see these two objects as seperate web [...]...