This post has been merged into the robots.txt and AI crawlers guide. Continue.