How do robots work?

Search engine robots, also known as spiders and crawlers are automated data fetching applications that are used to build the search engine's index (database) of websites. There are two ways to get a search engine spider to visit your site. The first is to submit the URL to the search engine via its URL submissions page. The second way is to get links from other websites that the search engine already has in its database.

When a search engine robot visits a site it checks for the presence of a robots.txt file (see the knowledge base for an explanation of robots.txt files). Once it knows which pages and areas are off-limits it will start to request pages from the site and feed the code back to the search engine database. The links from each page will be put in the queue of documents that the robot will request. The timing between requests and the number of documents requested will vary and are dependent on how the engineers have configured the robot to behave.

An optimised website will make the job of the robot as easy as possible by providing it with easy to follow links and a fully accessible site architecture.

Related Information