Google Premier Partner
Meta Business PartnerWhat is the X-Robots-Tag?
The X-Robots-Tag is an HTTP header that gives search engines instructions about indexing and displaying a file, such as noindex or nofollow. It works the same way as the meta robots tag, but it is sent in the server response. That means you can also use it for PDFs, images and other files without HTML.
Why is the X-Robots-Tag important?
- Works for any file type: a PDF has no head in which a meta tag could go.
- Many files at once: you set it in the server configuration for a whole pattern of files.
- Same instructions: you use values such as noindex, nofollow and noarchive.
Examples of the X-Robots-Tag
X-Robots-Tag: noindexX-Robots-Tag: googlebot: nofollowadd_header X-Robots-Tag noindex;in an nginx configuration
How do you use the X-Robots-Tag?
Set the header in your server configuration or hosting panel for the files you want to keep out of the index. Do not block those files in robots.txt, or Google will never see the header (technical SEO).