I changed my robots.txt file to exclude TurnitinBot but it still continues to come back?
If we re-requested the robots.txt file before each page request it would put a significantly larger load on servers and be wasteful of bandwidth. We get around this by caching robots.txt files. For versions Turnitinbot/1.4 and below, we cache the robots.txt file for 48 hours before we refresh our copy. As of version Turnitinbot/1.5, we dropped this value to 12 hours to better suit the needs of webmasters.
Related Questions
- I was an international student (F-1 visa) until October 1, and then my visa status was changed to H-1. Should I file my income tax return as a dual status or nonresident alien?
- I was an international student (F-1 VISA) until October 1, then my VISA status was changed to H-1. Should I file my income tax return as Dual Status or Nonresident Alien?
- I changed my robots.txt file to exclude TurnitinBot but it still continues to come back?
