Robots.txt

The robots.txt file is a text file that is added to the root of your website. This file contains a set of instructions for web crawlers, which can shield parts of your website from crawlers and search engines.