Robots Meta Tag
Robots meta tags have the same function as robots.txt files do. By adding these tags to web pages, you shiel them from crawlers and search engines. The robots meta tag enable you to give instructions specific to one web page, whereas robots.txt files cover entire (sections of) websites.
Aside from shielding pages from indexing, you can also use robots meta tags to clarify which hyperlinks should be followed. Under the standard directive, all links should be followed and refer value from one page to another. If you want something else to happen, you will need to give the appropriate instructions.
<meta name=”robots” value=”index, follow” />‘Index, follow’ is the standard variable for this tag. It defines whether or not the page can be included in the search results, indexed, and if all links on this page can be followed.
<meta name="robots" value="noindex, nofollow" />By adding the variable ‘noindex, nofollow’ to the robots meta tag, you can tell web crawlers not to index a page and not to follow links from this page.
Other possible variables to add to the robots meta tag are:
- NoImageIndex: keeps images from being included in search results.
- None: an abbreviation for ‘noindex, nofollow’.
- NoArchive: makes sure that the search results always display the most recent version of a web page.
- NoCache: an alternative to NoArchive for Microsoft crawlers.
- NoSnippet: keeps search engines from displaying snippets of your web page in search results.
- NoTranslate: keeps search engines from displaying translations of your web page in the search results.
- Unavailable_after: tells search engines that a web page will only be available up to a set date.
- NoYDir: keeps search engines from copying the page description from Yahoo.
- NoODP: keeps search engines from copying the page description from DMOZ.
You can combine almost all directives to instruct web crawlers to do as you see fit. NoYDir and NoODP have recently become redundant because Yahoo and DMOZ listings have been closed down.