LLM bot reference
Check AI Crawler Access for Top LLM Bots
Short answer: these nine user-agent tokens decide whether SearchGPT, Perplexity, Bing AI, GPT, Claude and the training crawlers behind them can read your site — allow the citation bots below to stay citable, and choose your training policy per crawler. The table mirrors exactly what the checker tests, so reference and results always agree.
| AI User-Agent | Class | What it does | Suggested robots.txt config |
|---|---|---|---|
OAI-SearchBotOpenAIoai-searchbot | AI Search & Citation | The crawler that powers SearchGPT and ChatGPT search results. | Allow: / — required for SearchGPT citations |
PerplexityBotPerplexityperplexitybot | AI Search & Citation | Fetches pages for Perplexity answers and API-backed products. | Allow: / — required for Perplexity answers |
BingbotMicrosoftbingbot | AI Search & Citation | The classic Bing crawler — also the fetch layer behind Bing AI / Copilot citations. | Allow: / — Bingbot feeds Bing AI & Copilot citations |
GPTBotOpenAIgptbot | Model Training | OpenAI's web crawler for model training and knowledge retrieval. | Allow: / or Disallow: / — your training policy |
ClaudeBotAnthropicclaudebot | Model Training | Anthropic's crawler for Claude model training and knowledge retrieval. | Allow: / or Disallow: / — your training policy |
Google-ExtendedGooglegoogle-extended | Model Training | The opt-out token for Google AI training, AI Overviews and Vertex AI grounding. | Disallow: / to opt out of AI Overviews & Vertex; Allow: / to stay included |
CCBotCommon Crawlccbot | Model Training | The open web corpus many LLM vendors train on, directly or indirectly. | Allow: / or Disallow: / — feeds many LLM training sets |
ChatGPT-UserOpenAIchatgpt-user | User Triggered | Fetches pages on-demand when a ChatGPT user browses or searches the web. | Allow: / — enables live ChatGPT browsing |
Claude-WebAnthropicclaude-web | User Triggered | Fetch layer for user-initiated actions in the Claude web and mobile apps. | Allow: / — enables live Claude browsing |
Suggested configs are GEO starting points — the snippet generator below turns your choice into copy-paste rules. Blocking training crawlers does not affect citation crawlers, and vice versa.
Why it matters
Why AI Crawlers (GPTBot, ClaudeBot, PerplexityBot) Matter for GEO
Short answer: a crawler that cannot read your site cannot cite it — robots.txt is the first gate of GEO, and every AI answer engine sits behind one of the user-agent tokens above.
When SearchGPT, Perplexity or Claude answers a question, the crawler has toread your page before it can cite it. A blanket Disallow: / on an AI crawler makes your site effectively invisible to that product — no citation, no referral traffic, no brand mentions. This is why GEO audits start with robots.txt rather than metadata.
An empty or missing robots.txt means allow by default — but in practice that is fragile. AI vendors are increasingly conservative about crawling sites that never send an explicit signal, and your own wildcard rules from a pre-AI era may quietly block every crawler on this list. Explicit per-crawler rules turn an accident into a decision.
robots.txt is only one layer: noai /noimageai meta tags protect individual pages, and a well-formed sitemap.xml tells answer engines which URLs matter most. Check the robots layer first — everything else is wasted if the crawler never gets in.
How it works
Three Steps, One Verdict Per Crawler
1. Enter a domain
The AIGEOKit edge proxy fetches
robots.txtfor you — browsers cannot read third-party robots.txt files directly due to CORS.2. Rules are parsed
User-agent groups, wildcards (
*,$) and Allow/Disallow precedence are evaluated the way Googlebot evaluates them (RFC 9309), per AI crawler token.3. Verdicts and a fix
Each crawler gets an Allowed / Blocked / Default badge with the rule that decides, plus a ready-to-paste block that makes your AI access explicit.
🔒 Privacy: robots.txt is public data and the proxy stores nothing. In paste mode nothing leaves your browser at all.
FAQ
Robots.txt & AI Crawlers, Explained
What do Allowed, Blocked and Default mean?
Allowed means this crawler has an explicit rule and can read your site root. Blocked means a rule denies it access to your whole site — usually a Disallow: / entry. Default means no rule mentions this crawler, so it inherits your User-agent: * rules (or, without a wildcard group, is simply allowed by default): nothing blocks it today, but no AI vendor sees an explicit signal either, which is why adding an explicit rule is the safest GEO move.
Do AI crawlers actually respect robots.txt?
The major vendors say they do: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-Web), Microsoft (Bingbot), Perplexity, Google (Google-Extended) and Common Crawl (CCBot) all document robots.txt support. Robots.txt is a request, not a legal guarantee — some smaller scrapers ignore it — so pair explicit allow rules with noai/noimageai meta tags on pages you want protected.
I never added AI crawler rules — why does the checker flag issues?
Because robots.txt groups that do not mention a crawler still apply to it when they use the wildcard User-agent: * token. A Disallow: / you added years ago for other reasons blocks GPTBot, PerplexityBot and every other AI crawler too. The checker shows a "via User-agent: *" source chip whenever that happens, and the suggestion panel generates the explicit allow rules that fix it.
I block GPTBot — does that keep my content out of SearchGPT?
Not by itself. SearchGPT and ChatGPT search use OAI-SearchBot, while on-demand browsing in ChatGPT uses ChatGPT-User — all three are separate user-agent tokens with separate rules. Blocking GPTBot mostly stops model-training use; add rules for the other two OpenAI tokens if you want full control. The same split exists at Perplexity (PerplexityBot vs Perplexity-User) and Anthropic (ClaudeBot vs Claude-Web).
What is the difference between blocking AI crawlers and blocking AI features?
Blocking a crawler stops its fetches entirely — and when the crawler doubles as the search indexer, as Bingbot does, the AI answers on top of it disappear too. Opt-out tokens like Google-Extended exist so you can keep normal Googlebot indexing while telling Google AI features such as AI Overviews that your content is off-limits. This checker treats every token as a separate row, so you can see exactly which lever you have pulled.
Why does a URL check need a server, and is my data safe?
Robots.txt is public data — your domain is fetched for the same reason a search engine fetches it. Browsers cannot read a third-party robots.txt directly (CORS), so AIGEOKit's edge function fetches the file for you; nothing is logged or stored. If you prefer zero network traffic, switch to paste mode: parsing then happens 100% in your browser.
Next steps
Keep Optimizing for AI Search
JSON-LD Schema Generator & GEO Checklist
Crawler access is step one. Generate Article, FAQ and Organization schema with a live GEO Readiness score.
Open Schema Generator →llms.txt Generator & Validator
Crawler access is half the story — generate a curated llms.txt so LLM crawlers know which of your pages are worth reading and citing.
Open llms.txt Builder →GEO Citation Analyzer
Want to check if AI can extract your content? Run a GEO Citation Analysis to see how answer engines parse and attribute your pages.
Browse all tools →