What is a robots.txt file?
A robots.txt file is a simple text file that is placed in the root directory of your website. The purpose of the file is to let search engine robots know which parts of the site they are allowed, and more importantly, not allowed to index.
The standard behaviour is for the search engine robot to request the file each time it pays a visit to your site. Once it has read the file it knows what it is allowed and not allowed to add to the search engine database. If the robots.txt file doesn't exist the search engine will assume that the entire site is okay to be spidered and indexed.
The file is made up of a number of entries in the following format:
User-agent: *
Disallow: /
This robots configuration will disallow all robots access to all areas of the site.
For more information on the robots protocol visit robotstxt.org. To see examples of robots files simply send a request for www.somedomainname.com/robots.txt.
Back to the Knowledge Base
|
|
Related Information
 Search engine optimisation
Search engine optimisation can be a cost effective way of driving more traffic to your website. Search Engine Optimisation
Pay per click management
Kolberg can offer a range of pay per click services that can put your website at the top of some of the most popular search engines. Pay Per Click Management
Pay for inclusion
Pay for inclusion is the easiest way into some of the major search engines Paid Inclusion Strategies
Authority development
Generate more qualified traffic by turning your website into an industry authority. Authority Development
|
