Robots.txt

A robots.txt file contains instructions for compliant web crawlers about which parts of a website they may crawl. It does not reliably prevent a URL from being indexed and should not be used to protect confidential information.