User agent

What is a user agent?

A user agent is a line of text in an HTTP request that a browser, search engine bot or other program uses to identify itself to a web server. The server can then see who is visiting and, for example, adjust the output or restrict access. For search engines, the user agent is the name of the crawler.

Why is a user agent important?

• Rules per bot: in robots.txt you decide per user agent what may be crawled.
• Insight into visits: your server logs show which crawlers visit your site.
• Beware of fake bots: a user agent is easy to fake, so verify real Googlebot visits through Google’s own methods.

Examples of a user agent

• User-agent: Googlebot
• User-agent: Bingbot
• User-agent: GPTBot (OpenAI’s crawler)

How do you use a user agent?

Use user agents in your robots.txt to allow or block specific crawlers, and review your server logs to see which bots visit your site most often (technical SEO).