Robots Meta Tag

A robots meta tag provides search engines with instructions about the indexing and presentation of an individual web page. Directives such as noindex, nofollow, nosnippet and noimageindex indicate how search engines may process the page and its content. A crawler must be able to access the page to read its robots meta tag.

  • <meta name="robots" content="index, follow"/>

The index, follow directives allow search engines to index the page and use its links to discover other pages. This is the default behaviour and therefore does not normally need to be specified explicitly.

  • <meta name="robots" content="noindex, nofollow"/>

The noindex, nofollow directives ask search engines not to show the page in search results and not to follow the links on the page. Search engines may interpret directives differently; Google also treats certain link attributes and instructions as hints.

Other possible robots meta tag directives include:

  • noimageindex: asks search engines not to index images on the page.
  • none: equivalent to noindex, nofollow.
  • nosnippet: prevents a snippet from being shown for the page in search results.
  • notranslate: asks search engines not to offer a translated version of the page in search results.
  • unavailable_after: asks search engines not to show the page in search results after a specified date and time.

Multiple compatible directives can be combined to control how web crawlers process the page.