How Do You Prevent Duplicate Content With Effective Use Of The Robots.Txt And Robots Meta Tag?
Duplicate content is one of the problems that we regularly come across as part of the search engine optimization services we offer. If the search engines determine your site contains similar content, this may result in penalties and even exclusion from the search engines. Fortunately it s a problem that is easily rectified. Your primary weapon of choice against duplicate content can be found within The Robot Exclusion Protocol which has now been adopted by all the major search engines. There are two ways to control how the search engine spiders index your site. 1. The Robot Exclusion File or robots.txt and 2. The Robots Tag The Robots Exclusion File (Robots.txt) This is a simple text file that can be created in Notepad. Once created you must upload the file into the root directory of your website e.g. [http://www.yourwebsite.com/robots.txt]. Before a search engine spider indexes your website they look for this file which tells them exactly how to index your site s content. The use of t
