Changing the User-Agent and Ethics
User-Agent Guide
The User-Agent is a value set by the client, so it can be changed freely. It's easily set with browser developer tools, extensions, HTTP client libraries, and the like.
Legitimate Uses
- QA that tests rendering across multiple browsers and devices
- Reproducing a specific environment in automated tests
- Setting a UA so that your own crawler honestly identifies itself
Uses to Avoid
- Impersonating a search engine bot to bypass paid content or access restrictions
- Mass scraping while ignoring robots.txt and terms of service
- Disguising identity to evade fees or policies
If You're Building a Crawler
It's good practice to put an identifiable UA and a contact (or an informational URL) in your own crawler. Example: MyBot/1.0 (+https://example.com/bot). This way, a site operator can reach you if a problem arises.
What is technically possible and what is ethically permissible are different things. Changing the UA is a powerful tool, so use it responsibly.