What are robots and spiders?
Search engines such as Google and Yahoo! use what is called ‘robots’ or ‘spiders’ to visit pages on the internet and then automatically add them to their search database. Many people even add their sites manually rather than wait for a robot or spider to visit their web site. When you put a web page on your web server, it can take some time for your site to show up in their search engine. Once the page is entered into their database, however, it can take a long time also for the page to be removed should the page move or be taken off the server.
