AI Crawler Detector

What this detector checks

The tool matches known user-agent tokens for AI crawlers and AI-related web fetchers including GPTBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent, Amazonbot and Omgilibot. It also flags regular search crawlers such as Googlebot and Bingbot so you can separate AI access from normal indexing.

Where to find user agents

Check your hosting access logs, CDN logs, WAF events or analytics bot reports. In Apache and nginx combined logs, the user-agent is usually the quoted string near the end of each line. You can paste complete log rows here; the detector scans the whole line.

Frequently asked questions

Can a scraper fake its user-agent?

Yes. User-agent detection is useful for honest or self-identifying bots, but it is not a security boundary. Use server logs, IP reputation, rate limits and WAF rules for enforcement.

Does robots.txt block all AI crawlers?

No. Reputable crawlers may honor robots.txt, but some scrapers ignore it. Use a robots.txt generator for crawler signals and server-side controls for real blocking.

What should I do with unknown lines?

Look for repeated access patterns, unusual paths, high request rates and missing referrers. Unknown does not mean safe; it only means the user-agent did not match this known-bot list.