Most robots.txt files were written before ChatGPT existed. The tool tests yours against 25+ modern AI bots - OpenAI, Anthropic, Perplexity, Google Gemini, Apple, ByteDance, Meta, Cohere, Mistral, You.com, Amazon, Common Crawl and more - and flags the strategic mistakes most sites do not know they are making.
Run a check to see the score.
One opinionated template - allows search + citation-fetch bots (so you appear in ChatGPT search, Perplexity answers, Claude search), blocks bulk training-only bots (so your content does not enter training datasets without referral payback). Other defensible stances exist - this is mine.
| Bot | Vendor | Role | Status |
|---|
Bot weights mapped to citation impact for
Citation crawlers (OAI-SearchBot / PerplexityBot / Claude-SearchBot) weigh far more than training-only crawlers (GPTBot / Google-Extended). The score reflects who actually drives traffic to each engine below.
ChatGPT
Claude
Perplexity
Gemini
Copilot
Llama
Grok
Mistral
DeepSeek
GPTBot trains the underlying model. OAI-SearchBot indexes the live ChatGPT search results. Blocking one does not block the other. Most sites block both by accident and wonder why they get no AI search traffic.
Blocking Google-Extended means Gemini cannot use your content for training or grounding. It does not affect your standard Google search rankings. The two bots are independent controls and most sites do not realise.
Common Crawl is the dataset every major LLM trains on. Blocking CCBot means no LLM has any baseline knowledge of your brand, which collapses the chance of being cited in unfocused queries.
Some publishers have blocked all AI bots in protest. The cost is invisibility in AI search - and reported conversion rates from AI search referrals run 4 to 23x Google organic depending on vertical (Seer, Semrush, Ahrefs studies). The calculation has changed.
Allowing the right bots gets you crawled. Getting cited needs entity work, schema and content built for AI extraction. The tool handles the first; the services below handle the rest.
Right bot policy plus the entity graph, schema and content work that actually gets cited once bots can crawl. The full GEO programme.
See the GEO service SEO ManagementFor sites where robots.txt is one technical issue among many. Monthly senior retainer that covers technical SEO, content, links and AI search together rather than in silos.
See SEO Management Technical SEOFor sites where crawl rules, blocked paths, sitemap bloat or canonicals are stopping useful pages from being found and trusted.
See Technical SEO Digital StrategyPublishers, regulated businesses and enterprise brands need a deliberate stance on AI training, citation and licensing. Strategy work that makes the trade-offs explicit instead of accidentally choosing them.
See Digital StrategyThe right robots.txt depends on what your business does. Publishers, e-commerce, B2B SaaS and brand sites all want different things. 30-min call to walk through the right policy for yours.