Does the Exalead robot respect the rules recorded in a robots.txt file and the robots META tag?
Yes. Exalead’s Exabot robot fully complies with robots.txt and robots META tag standards. Please visit robotstxt.org for more information regarding these specifications. Exalead also supports the special characters * and $, which were not included in the initial specification. Robots.txt standards: • To prevent indexing of pages from a particular directory (for example, football), enter the following in your robots.txt file: User-agent: Exabot Disallow: football • To prevent indexing of a particular file type (.gif, par exemple), enter the following in your robots.txt file: User-agent: Exabot Disallow: *.gif$ • To prevent indexing of dynamic pages, enter the following in your robots.txt file: User-agent: Exabot Disallow: *?
