How to Identify Bots and Crawlers Safely

Search engine bots, social preview bots, and AI crawlers mostly use a User-Agent that identifies themselves. For example, Googlebot includes a token like Googlebot/2.1, and GPTBot includes one like GPTBot/1.1.

Why Judging by UA Alone Is Risky

Anyone can forge a User-Agent. If a malicious scraper impersonates Googlebot to gain access, a server that trusts only the UA string can't filter it out.

Recommended Verification Procedure

To trust a legitimate bot, check the following together.

• First, check whether the UA contains a known bot token
• Do a reverse DNS lookup on the accessing IP to check whether it belongs to an official domain such as googlebot.com
• Confirm that result again with a forward DNS lookup (reverse + forward match)
• Cross-check against the IP ranges provided in official documentation

Relationship with robots.txt

Polite bots respect robots.txt. To control crawling, combining robots.txt with server-side IP verification is more robust than blocking by UA.

AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) also mostly follow robots.txt rules, so if you want to block training crawls, you can Disallow the relevant UA in robots.txt.