The HTML robots meta tag provides instructions to search engine crawlers about how they should handle a webpage. Depending on the directives you specify, you can request that a page not be indexed or ask crawlers not to follow links on that page.
The robots meta tag is especially useful for website owners who want to manage which pages appear in search results. However, it is important to use it correctly because an incorrect directive can prevent an important page from being indexed.
In this tutorial, you will learn how the HTML robots meta tag works, understand its common directives, and see practical examples.
What Is the HTML Robots Meta Tag?
The robots meta tag is an HTML metadata element that gives crawling instructions to search engines that support it. It is placed inside the <head> section of an HTML document.
For example:
<meta name="robots" content="index, follow">This example explicitly requests that compliant crawlers index the page and follow its links. These are generally the default behaviors, so the tag is often unnecessary.
You can learn more about placing metadata correctly in our HTML Head tutorial.
HTML Robots Meta Tag Syntax
The basic syntax is:
<meta name="robots" content="directives">The name="robots" attribute identifies the intended audience as search engine robots, while content contains one or more directives separated by commas.
For example:
<meta name="robots" content="noindex, nofollow">This requests that compliant crawlers not index the page and not follow its links.
Common Robots Meta Tag Directives
Here are some commonly used directives:
| Directive | Meaning |
|---|---|
index | Allows indexing of the page |
noindex | Requests that the page not be indexed |
follow | Allows links on the page to be followed |
nofollow | Requests that links on the page not be followed |
nosnippet | Requests that no text snippet or video preview be shown in search results |
noarchive | Legacy directive with limited or no support in some major search engines |
The exact behavior depends on the search engine. For Google, index and follow are generally the default, while noindex is commonly used when a page should not appear in search results.
How to Prevent a Page from Being Indexed
If you do not want a particular page to appear in search results, you can use the noindex directive.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="robots" content="noindex">
<title>Private Preview Page</title>
</head>
<body>
<h1>Preview Page</h1>
<p>This page requests that search engines not index it.</p>
</body>
</html>The noindex directive asks compliant search engines not to include the page in their search results.
Important: This is not a security feature. It does not prevent visitors from opening the page, and crawlers must be able to access the page to read the meta tag. Use authentication or access controls for confidential content.
How to Control Link Following
You can use nofollow when you want to request that search engine crawlers not follow links on a page.
<meta name="robots" content="noindex, nofollow">Here, noindex requests that the page not be indexed, while nofollow requests that links on the page not be followed.
These directives do not remove the links from the webpage. Visitors can still click them normally.
Robots Meta Tag vs. Robots.txt
The robots meta tag and robots.txt file serve different purposes.
| Feature | Robots Meta Tag | Robots.txt |
|---|---|---|
| Location | Inside the HTML <head> | Separate file at the site’s root |
| Main purpose | Provides indexing and link-handling directives | Controls crawler access to URL paths |
| Example | <meta name="robots" content="noindex"> | Disallow: /preview/ |
| Requires page access? | Yes, crawlers must access the page to read it | Crawler rules can prevent access to a URL path |
If a page is blocked by robots.txt, a crawler may not be able to read its noindex meta tag. Therefore, do not block a page from crawling when you need a search engine to discover and process its noindex directive.
Google-Specific Robots Meta Tags
You can also target a specific crawler by using its name.
<meta name="googlebot" content="noindex">This directive targets Google’s general web crawler. The broader robots meta tag applies to supported crawlers generally.
Use crawler-specific directives only when you need different instructions for different crawlers.
Best Practices for Robots Meta Tags
- Use
noindexonly on pages that should not appear in search results. - Check important pages to ensure they are not accidentally marked
noindex. - Remember that
index, followis normally unnecessary because these are default behaviors. - Do not use robots meta tags to protect sensitive information.
- Make sure crawlers can access pages when they need to read indexing directives.
- Test your settings before applying them across an entire website.
Frequently Asked Questions
Is the robots meta tag required?
No. If you do not add a robots meta tag, search engines generally use their default behavior, subject to other crawling and indexing rules.
Does noindex remove a page immediately?
No. Search engines need to crawl and process the directive. The timing depends on the crawler and the page.
Can I use noindex and follow together?
Yes. For example, content="noindex, follow" requests that the page not be indexed while allowing crawlers to follow its links. Search engines may handle link-following behavior according to their own policies.
Does robots.txt prevent indexing?
Robots.txt controls crawling, not indexing directly. A blocked URL can sometimes still appear in search results if it is discovered through other sources. Use the appropriate method for your goal.
Should I use noindex on every page that is not important?
No. Use it selectively. Important articles, category pages, and landing pages should remain eligible for indexing when they are intended to attract search traffic.