What is User Agent?

Key Takeaways

Every request to a website carries a user agent string that names the program making it.
Crawlers use a published user agent name so site owners can recognize them and set rules for them.
robots.txt rules are matched to the user agent name.
The string can be faked, so it identifies a program but does not prove who is behind it.

A web browser sends a user agent that names the browser and system. A crawler sends one that names the crawler and often a link to its documentation.

Search engines and AI companies list their user agents on their own help pages, with what each one is used for.

Because anyone can send any string, a site that needs certainty can check the network address of the request against the ranges the vendor publishes.

Is a user agent the same as a browser?

No. A browser has a user agent, and so does every crawler and automated tool.

Why does robots.txt use user agents?

So a rule can apply to one crawler without affecting others.

Can a user agent be faked?

Yes. Verify important cases using the network ranges the vendor publishes.