Updated
Updated What changed on
- Corrected the heading structure. Two headings naming the meanings of AI bot detection were h4 tags placed directly after the h1, skipping two levels. They are now h2. No wording changed.
"AI bot detection" splits into two jobs: crawler policy and text detection.
Half the people typing this query want to stop GPTBot, ClaudeBot and PerplexityBot from scraping their site. The other half want to know whether a paragraph was written by ChatGPT. Same words, totally different tools. This page covers both, picks honest defaults for each, and tells you which one you actually need.
AI crawler / agent detection
Spotting and managing which AI bots crawl your site. GPTBot, ClaudeBot, PerplexityBot, autonomous agents. Decision is allow, block, or monetise.
Top picks: Cloudflare AI Audit, Known Agents, TollBit, HUMAN AgenticTrust.
AI-generated text detection
Working out whether a piece of writing came from a human or an AI. Used by editors, lecturers, publishers, content QA teams.
Top picks: GPTZero, Originality.ai, Winston AI, Copyleaks.
- AI agent traffic jumped 7,851% during 2025, per HUMAN Security's 2026 traffic report. The dashboards moved from optional to default-on as a result.
- Dark Visitors rebranded to Known Agents in 2025, with the AI-agent tracking and policy templates folded into the new brand.
- TollBit's publisher-side licensing marketplace matured. AI buyers now pay per-transaction fees rather than blocking being the only option for premium content.
- Cloudflare AI Audit moved from beta to general availability for all Cloudflare-fronted sites.
- HUMAN Security launched AgenticTrust, aimed at the new problem of separating sanctioned AI agents from unsanctioned ones rather than blocking AI wholesale.
The 30-second pick
On Cloudflare and just want a default: Cloudflare AI Audit, already included, no extra spend. Want visibility into who is crawling: Known Agents (formerly Dark Visitors). Publisher monetising crawler access: TollBit. Checking whether text was written by AI: GPTZero for one-offs, Originality.ai for content operations, Winston AI for high-stakes review. Free baseline check before paying for anything: the AI Bot Robots.txt Tester on this site.
Key takeaways
- Two different categories share the phrase "AI bot detection". Pick the bucket first, the tool second.
- For crawler policy, start at the CDN layer. If you are on Cloudflare, that is the cheapest and fastest control plane.
- For text detection, no tool hits 100%. Premium tiers reach mid-80s accuracy on independent testing. Treat any score as a trigger for human review, not a verdict.
- If you are a UK small business with no immediate bot-abuse problem, robots.txt plus log review is enough. Tools matter once cost or risk justifies the licence.
Which "AI bot detection" do you actually need?
The query is ambiguous on purpose - both meanings have grown together. Use this table to land on the right side before you start comparing tools.
| If your question is... | You need... | Start with... |
|---|---|---|
| "Is GPTBot scraping my site?" | Crawler detection | Cloudflare AI Audit or Known Agents |
| "Should I let Perplexity crawl us?" | Crawler policy | AI Bot Robots.txt Tester (free), then Cloudflare |
| "Can we charge AI engines for access?" | Crawler monetisation | TollBit |
| "Did a freelancer just send us ChatGPT output?" | Text detection | GPTZero or QuillBot (free check) |
| "Are we publishing AI-written content at scale by accident?" | Text detection at scale | Originality.ai or Copyleaks |
| "This is an academic / regulated context." | High-stakes text detection | Winston AI (premium tier) |
| "Bots are filling our signup forms with junk." | Lead / abuse bot detection | Fingerprint, DataDome or Arkose Labs |
| "We want sanctioned AI agents in, scrapers out." | Agentic governance | HUMAN AgenticTrust |
13 tools for managing AI crawlers and agents.
These are for the people asking "who is crawling my site and what do I do about it". Ordered by how often I actually recommend each one.
| Tool | Pricing | Best for |
|---|---|---|
| Cloudflare AI Audit | Included with Cloudflare | Sites already on Cloudflare |
| Known Agents (formerly Dark Visitors) | Free + paid (Individual $29, Startup $299) | Tracking which AI agents access your content |
| TollBit | Free for publishers | Publishers monetising AI crawler access |
| HUMAN Security (AgenticTrust) | Enterprise | Telling good AI agents from bad ones at scale |
| DataDome | From $3,690/month (Business tier) | Sites already losing money to bot abuse |
| Akamai Bot Manager | Enterprise | High-traffic publishers and regulated industries |
| Fingerprint | Pro Plus $99/month, Enterprise custom | Developers who want bot detection as an API |
| Radware Bot Manager | Enterprise (30-day trial) | Enterprise sites that want intent-based scoring |
| Imperva Advanced Bot Protection | Enterprise | Security teams wanting bot policy inside a broader WAF |
| Arkose Labs | Enterprise (demo required) | Login and signup flows under sustained attack |
| CrowdSec | Free + paid tiers | Open-source teams that want shared threat data |
| AI Bot Robots.txt Tester (Jason Burns tools) | Free | Quick check of your current bot policy |
| Manual robots.txt + log monitoring | Free | Small sites with no immediate need for a tool |
1Cloudflare AI Audit
Included with CloudflareBest for: Sites already on Cloudflare
Built into Cloudflare. Shows which AI bots have crawled, lets you allow or block per bot, and feeds the AI Audit dashboard. Default choice if Cloudflare is already in your stack. Cloudflare's own 2026 traffic report flagged a 7,851% jump in AI agent traffic during 2025, which is what made this dashboard go from beta to general availability.
2Known Agents (formerly Dark Visitors)
Free + paid (Individual $29, Startup $299)Best for: Tracking which AI agents access your content
Specialist tool that tracks AI agents (GPTBot, ClaudeBot, PerplexityBot, etc) hitting your site. Generates robots.txt suggestions and policy templates. Rebranded from Dark Visitors to Known Agents in 2025.
3TollBit
Free for publishersBest for: Publishers monetising AI crawler access
Free for publishers. AI buyers pay per-transaction fees through TollBit's marketplace. Built around licensing rather than blocking. Right tool for paywalled or premium content sites.
4HUMAN Security (AgenticTrust)
EnterpriseBest for: Telling good AI agents from bad ones at scale
HUMAN (formerly PerimeterX) launched AgenticTrust specifically to separate sanctioned AI agents from unsanctioned ones. Strong fit if your problem is not "block AI" but "let the right AI in, block the rest" - e.g. agentic commerce flows.
5DataDome
From $3,690/month (Business tier)Best for: Sites already losing money to bot abuse
AI-driven detection with strong API and mobile coverage. Public Fingerprint pricing teardown puts Business at $3,690/mo, Corporate $6,490/mo, Enterprise $8,590/mo. Not the right call until you can show bot traffic is costing more than the licence.
6Akamai Bot Manager
EnterpriseBest for: High-traffic publishers and regulated industries
Behavioural modelling on top of Akamai's network telemetry. Sensible when you already use Akamai for CDN; usually overkill otherwise.
7Fingerprint
Pro Plus $99/month, Enterprise customBest for: Developers who want bot detection as an API
Device-and-browser fingerprinting with a clean JS agent. Pro Plus at $99/mo for 20k API calls makes it the cheapest entry point on this list if you can wire it into your own stack. CAPTCHA-free, which matters for conversion-sensitive forms.
8Radware Bot Manager
Enterprise (30-day trial)Best for: Enterprise sites that want intent-based scoring
Three-layer model: preemptive IP blocks, AI behavioural detection, then granular mitigation. Strong fit if you already run a Radware WAF.
9Imperva Advanced Bot Protection
EnterpriseBest for: Security teams wanting bot policy inside a broader WAF
Bot management is one feature of a broader security platform. Right fit when AI bot policy is part of a larger security review, not a standalone need.
10Arkose Labs
Enterprise (demo required)Best for: Login and signup flows under sustained attack
Dynamic challenge-based mitigation. The dynamic interception layer is the differentiator - it wastes attackers' compute rather than just blocking. Useful for high-value account creation.
11CrowdSec
Free + paid tiersBest for: Open-source teams that want shared threat data
Open-source detection engine with community-shared signatures. Includes AI bot rules. Lower-cost fit for technical teams that want transparency.
12AI Bot Robots.txt Tester (Jason Burns tools)
FreeBest for: Quick check of your current bot policy
Free tool on this site. Checks whether each major AI bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more) can crawl your site. Useful as a baseline before deciding whether you need a paid tool.
13Manual robots.txt + log monitoring
FreeBest for: Small sites with no immediate need for a tool
For most small sites, a well-configured robots.txt plus periodic log review is enough. Tools matter when scale or commercial stakes justify them, not on day one for a small business site.
5 tools for detecting AI-written content.
These are for editors, lecturers, publishers and content QA teams. None of them are perfect. Independent testing in 2026 puts the best paid tiers in the mid-80s percent accuracy and free tiers in the mid-to-high 70s. Use the score as a trigger for human review, not as a verdict.
| Tool | Pricing | Best for |
|---|---|---|
| GPTZero | Free + paid | Education, publishing, editorial review |
| Originality.ai | From $0.01 per 100 words (Base plan) | Content teams, SEO agencies, publishers |
| Winston AI | From $12/month | High-stakes review where false positives have cost |
| Copyleaks | From $7.99/month (small plan) | Multilingual content and code review |
| QuillBot AI Detector | Free | Free one-off checks for individuals |
1GPTZero
Free + paidBest for: Education, publishing, editorial review
Independent testing in 2026 still ranks GPTZero among the most accurate consumer-grade detectors, with sentence-level highlighting and a published accuracy methodology. Defaults to a probability score plus highlighted likely-AI sentences.
2Originality.ai
From $0.01 per 100 words (Base plan)Best for: Content teams, SEO agencies, publishers
Site-wide scanning, an API, and version control over what was AI-edited vs AI-written. Built for content operations rather than one-off checks. Pricing is usage-based, which keeps cost honest at low volume.
3Winston AI
From $12/monthBest for: High-stakes review where false positives have cost
Strong reputation in education and publishing. The HUMN-1 certification programme lets a site display a verification badge tied to Winston's scoring, which some publishers use as a trust signal.
4Copyleaks
From $7.99/month (small plan)Best for: Multilingual content and code review
One of the few detectors that handles AI-generated source code, not just prose. Multilingual coverage is wider than most. API-first, so it slots into CMS workflows.
5QuillBot AI Detector
FreeBest for: Free one-off checks for individuals
Free, with a sentence-by-sentence breakdown that separates "AI-written" from "human + AI-edited". Useful baseline. Independent tests put accuracy around the high-70s percent range, so not the right tool for high-stakes decisions on its own.
What these tools actually look at.
Crawler detection signals
- User-agent string - the cheapest signal. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Meta-ExternalAgent all identify themselves. Spoofable, so used in combination with the next two.
- IP range and reverse DNS - cross-checks the user-agent against published IP ranges. OpenAI, Anthropic and Google publish theirs.
- TLS fingerprint and HTTP/2 behaviour - what Cloudflare and Fingerprint use to catch bots pretending to be browsers.
- Request patterns - rate, depth, paths visited. Scrapers don't browse like humans.
Text detection signals
- Perplexity - how predictable the word choices are. AI text tends to be too predictable.
- Burstiness - variation in sentence length. Human writing varies; AI writing levels out.
- Statistical fingerprints - patterns specific to GPT-4, Claude, Gemini, etc. Tools train on labelled corpora and update as models update.
- Stylometric features - lexical diversity, punctuation patterns, sentence openings.
Both categories share the same weakness: as the AI models improve, the detection signal degrades. Detection vendors are in a permanent arms race with the model vendors. Treat any tool's "99% accurate" marketing claim with appropriate suspicion.
What clients ask about AI bot detection.
What does "AI bot detection" actually mean?
Two completely different things. Crawler / agent detection spots when GPTBot, ClaudeBot, PerplexityBot or autonomous AI agents are hitting your site, and lets you allow, block or monetise that access. Cloudflare AI Audit, Known Agents and TollBit live here. AI-generated content detection tries to work out whether a piece of text was written by a human or an AI. GPTZero, Originality.ai and Winston AI live here. The two share the search query but solve different problems.
Which "AI bot detection" do I need?
If the question is "who is crawling my site and should I let them", you need crawler management. If the question is "did a human write this", you need a content detector. Marketing and SEO teams almost always want category one. Editors, lecturers and publishers almost always want category two. Sometimes both, in which case start with the question that has the bigger commercial cost attached.
Should I block AI bots or let them in?
Depends on the business model. For most marketing sites, allow AI bots - they drive citation in AI engine answers, which is becoming a real source of qualified traffic. For paywalled publishers or licensable content, blocking or charging through TollBit makes sense. Default is allow unless you have a specific reason to block.
How do I see which AI bots are crawling my site?
Server logs show user-agent strings. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Meta-ExternalAgent and others identify themselves clearly. Use the free AI Bot Robots.txt Tester for a quick policy check, Known Agents for ongoing visibility, or Cloudflare AI Audit if you already terminate at Cloudflare.
Is Cloudflare AI bot blocking enough?
For most sites already on Cloudflare, yes. The AI Audit dashboard and per-bot allow/block controls cover the common needs. Move to a specialist tool only if Cloudflare lacks something specific - usually publisher monetisation (TollBit), enterprise scraping defence (DataDome, Akamai) or agentic-traffic governance (HUMAN AgenticTrust).
Do AI bots respect robots.txt?
OpenAI, Anthropic and Google all publicly commit to honouring robots.txt for their named user agents. Perplexity has documented PerplexityBot and the robots.txt directive, but Cloudflare research in 2024 documented stealth crawling on sites that had blocked it - put Perplexity in the "mostly honours, with caveats" bucket. Smaller or less reputable AI crawlers may not respect robots.txt at all. If the content is worth scraping and reputation does not deter, a CDN-level block is more reliable than robots.txt alone.
How accurate are AI-text detectors?
Mid-to-high 70s percent accuracy is typical for the free tools (QuillBot, ZeroGPT, Scribbr free). Paid premium tiers (Scribbr premium, Winston AI, Originality.ai) score in the mid-80s on independent testing. None hit 100%. False positives are most common with edited human writing, ESL text and short samples. Treat any detector output as a signal that triggers a human review, not as a verdict.
What happens if I block all AI bots?
You disappear from AI citation pools. ChatGPT, Perplexity, Claude, Gemini and AI Overviews can no longer read or reference your content. For some publishers this is the right call. For most marketing and B2B sites, it removes a growing channel of qualified traffic.
- HUMAN Security - 2026 State of AI Traffic report (7,851% AI agent traffic growth stat)
- Cloudflare AI Audit product page
- OpenAI bot documentation
- Anthropic ClaudeBot documentation
- Fingerprint - 6 bot detection tools comparison (DataDome pricing reference)
- Scribbr - 2026 AI detector accuracy testing
Jason Burns
Independent UK SEO, GEO and AI consultant. 17 years in search. Portfolio includes 3M, BlackRock, Unilever and E.ON. Owner of SEO Moves Ltd since 2014.
More about Jason →Not sure which bucket you are in?
Send the URL and a one-line description of the problem. I tell you whether it is a crawler-policy question or a text-detection question, and which tool gets you the answer fastest. Quick reply, no sales call.
Send the URL