What is the meta robots tag?
The meta robots tag is an HTML meta tag that gives crawlers page-level indexing and serving instructions. Its most common directives are noindex (keep the page out of search results), nofollow (do not follow links on this page), noarchive, and nosnippet. The same directives can be sent as an X-Robots-Tag HTTP header, which is how you apply them to PDFs and other non-HTML files.
Meta robots vs robots.txt
The meta robots tag is frequently confused with robots.txt, but they operate at different layers. robots.txt controls whether a crawler may fetch a URL; the meta robots tag controls what the crawler may do with a page it has fetched. A page blocked by robots.txt can never have its meta robots tag read, which is why a noindex inside a robots.txt-blocked page does nothing.
Directives combine. content="noindex, follow" keeps a page out of the index while letting PageRank flow through its links, the standard setting for paginated archives and filtered listings you want crawled but not indexed.
Example
Example
A faceted category page at /shoes?color=red&size=9 carries a meta robots tag of "noindex, follow". Google drops it from the index but still follows its links to the product pages, preserving discovery.
Frequently asked questions
What is the difference between meta robots and robots.txt?
robots.txt stops crawling; the meta robots tag instructs what to do after crawling. To deindex a page, it must remain crawlable so Google can see the noindex directive.
What does "noindex, follow" do?
It removes the page from search results while letting crawlers follow its links and pass equity. Google eventually treats long-standing noindex pages as nofollow too, so it is a transitional state, not permanent architecture.