Google Premier Partner
Meta Business PartnerWhat is robots.txt?
Robots.txt is a text file in the root of a website (example.com/robots.txt) in which you state which pages and folders crawlers may or may not visit. It follows the Robots Exclusion Protocol. Well-behaved crawlers such as Googlebot respect it, but robots.txt is not a security measure.
Why is robots.txt important?
- Crawler control: you keep crawlers away from unimportant pages, such as a shopping cart or internal search results.
- Less waste: this way your crawl budget goes to the pages that matter.
- Signpost to your sitemap: you can point crawlers to your sitemap inside the file.
Examples of robots.txt
User-agent: *applies to all crawlersDisallow: /cart/blocks the /cart/ folderSitemap: https://www.example.com/sitemap.xmlpoints to your sitemap
How do you use robots.txt?
Place a single file at /robots.txt and only block what crawlers really don’t need to see. Never block the CSS or JavaScript files needed to render your pages. Don’t use robots.txt to keep pages out of Google either: a blocked page can still show up in the results. For that you use a noindex tag. See technical SEO.