Robots meta tags are used to control how search engines and other web crawlers interact with your web pages. Here’s a detailed overview of the specifications for these meta tags:

Basic Syntax

The meta tag is placed within the <head> section of an HTML document. The basic syntax is:

HTML

<meta name="robots" content="value">

or for specific bots:

<meta name="googlebot" content="value">

Common Meta Tags Values

Index / Noindex:
- index: Allows the page to be indexed by search engines. This is the default behavior if not specified.
- noindex: Prevents the page from being indexed by search engines.
Follow / Nofollow:
- follow: Allows search engines to follow the links on the page. This is the default behavior if not specified.
- nofollow: Prevents search engines from following links on the page.
Archive / Noarchive:
- archive: Allows search engines to store a cached copy of the page.
- noarchive: Prevents search engines from storing a cached copy of the page.
Snippet / Nosnippet:
- snippet: Allows search engines to show snippets (e.g., meta descriptions, titles) in search results.
- nosnippet: Prevents search engines from showing snippets.
Images / Noimages:
- images: Allows search engines to index images on the page.
- noimages: Prevents search engines from indexing images.

Example Usage

Prevent indexing and following links:

<meta name="robots" content="noindex, nofollow">

Allow indexing but not following links:

<meta name="robots" content="index, nofollow">

Prevent caching but allow indexing and following:

<meta name="robots" content="index, follow, noarchive">

Prevent snippets from being shown:

<meta name="robots" content="nosnippet">

Specific Bots

Meta tags can also be applied to specific search engines or bots. For example:

Googlebot:

<meta name="googlebot" content="noindex, follow">

Bingbot:

<meta name="bingbot" content="noarchive">

Robots Meta Tag vs. Robots.txt

robots.txt: A file located in the root directory of a site that provides directives to web crawlers about which parts of the site can or cannot be accessed. It affects the entire site or specific directories, but not individual pages within a directory.

robots meta tag: Provides more granular control and can be applied to individual pages or sections within a site.

Notes

The meta robots tag is typically used in combination with other SEO techniques to manage how your content is crawled and indexed.
Always test your robots meta tags to ensure they are working as expected. Use tools like Google Search Console to check how Google is crawling and indexing your site.

By using these meta tags appropriately, you can better control how your site appears in search engine results and how search engines interact with your content.