Resources
AI Crawlers and you:
How to block or allow them using robots.txt
The age of AI is here, and it's actively reading, interpreting, and learning from your website. This presents every site owner with a critical choice: do you want to embrace AI discovery, or do you want to protect your content from being used in language model training?
Fortunately, you have control. Whether you want to optimize your site for AI visibility or block access entirely, the primary tool for the job is a simple text file: robots.txt. This guide will provide you with the user agents and templates you need to implement your AI strategy today.
Understanding Your Control Toolkit
What is robots.txt?
At its core, robots.txt is a public text file that lives on your website's server. It gives instructions to web crawlers — including AI bots — about which pages and files they are allowed or forbidden to request. It is the web's standard for crawler protocol. For a deep dive, refer to the official documentation from Google's Developer Center.
What about llms.txt and ai.txt?
These are newer, experimental files proposed to give more specific instructions to AI models. While robots.txt is the main "gatekeeper," you could use llms.txt to point AI models to content specifically optimized for them, creating a better-tailored experience. For now, the best practice is to ensure your robots.txt rules are clear, as it is the most widely respected standard.
Is this an all-or-nothing choice?
Not at all. You have granular control. You can choose to block a specific AI crawler while allowing others. You can also direct one bot to your main content while pointing another to a different, AI-optimized section of your site.
Important: Is this guaranteed to work?
robots.txt operates on an honor system. Think of it as a "No Trespassing" sign on your lawn, not a locked gate. Major operators such as Google and OpenAI say that their automated crawlers follow robots.txt. Some also say that fetches a user starts may not follow it. Less scrupulous bots may ignore it. It is your first and most important line of defense, but not a foolproof security measure.
Known AI & LLM Crawler User Agents
User agents for the major AI crawlers and search engine crawlers. Each purpose below comes from the operator's own page, linked in the Source column. Where no operator page exists, the table says so.
Last Updated: October 13, 2026
Which AI crawlers should you allow?
Start by splitting the list in two, because the two halves cost you completely different things. Training crawlers collect content that may end up in a foundation model. Blocking them is a decision about how your content is used. Retrieval crawlers are what let an assistant find, quote and link you when someone asks a question today. Blocking those can remove you from the answer.
Most site owners who say "block the AI bots" mean the first group and not the second. The two are
easy to conflate because vendors run both under similar names — OpenAI's
GPTBot collects content for training,
OAI-SearchBot surfaces sites in ChatGPT search.
The templates below give you all three positions: block everything, allow everything, or block
training while staying visible in AI search.
One correction worth making explicitly, because it is the most common misconception on this topic:
blocking Google-Extended
does not affect your inclusion or ranking in Google Search.
Google documents it as controlling Gemini training and grounding only. Blocking
Googlebot, on the other
hand, removes you from Search entirely.
Training Crawlers and Opt-out Tokens
Their operators say that the content they collect can be used to train AI models. Google-Extended and Applebot-Extended are not crawlers. They are tokens that Google and Apple read to decide how they use the content that Googlebot and Applebot collect.
| User-agent |
|---|
| GPTBot |
| ClaudeBot |
| meta-externalagent |
| Amazonbot |
| MistralAI-Training |
| Google-Extended |
| Applebot-Extended |
Retrieval Crawlers: Search and Answers
Their operators document them as crawlers for search results and answers, or as fetchers that visit a page when a user asks. PerplexityBot belongs here: Perplexity states that it is "not used to crawl content for AI foundation models."
| User-agent |
|---|
| OAI-SearchBot |
| ChatGPT-User |
| Claude-SearchBot |
| Claude-User |
| PerplexityBot |
| Perplexity-User |
| YouBot |
| meta-webindexer |
| Amzn-SearchBot |
| DuckAssistBot |
| MistralAI-Index |
Search Engine Crawlers
Crawlers for general search engines. Diffbot is here because Diffbot describes its crawler as building a general search engine, and says that it is not used for AI training.
| User-agent |
|---|
| bingbot |
| Googlebot |
| Applebot |
| Diffbot |
Datasets and Crawlers Without an Operator Page
Common Crawl publishes an open dataset. ByteDance and Cohere publish no page that names these tokens, so this page states no purpose for them.
| User-agent |
|---|
| CCBot |
| cohere-ai |
| Bytespider |
Our Own Crawler
robots.txt. The one exception to on-demand use is a research study that reads only robots.txt, llms.txt and ai.txt. PaceghostBot documentation → robots.txt Templates for AI Governance
Template 1: Block All Known AI Crawlers
Use this template if your goal is to ask the major AI crawlers not to use your content for training or answers.
Know what this costs. This template blocks both halves — training crawlers and retrieval crawlers — so your content stays out of model training AND out of AI answers. That is a legitimate choice, but it is a bigger one than most people intend when they say "block the AI bots." If you want to stop the training without disappearing from ChatGPT, Claude and Perplexity, use Template 3 instead. Some fetches that a user starts can still reach you: Perplexity says Perplexity-User "generally ignores robots.txt rules", and OpenAI says robots.txt rules "may not apply" to ChatGPT-User.
# === BLOCK ALL MAJOR AI CRAWLERS ===
# Template provided by paceghost.io - The AI Readiness Platform
# Last updated: October 13, 2026
# == Training crawlers and opt-out tokens ==
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: MistralAI-Training
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# == Datasets and crawlers without an operator page ==
User-agent: CCBot
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: Bytespider
Disallow: /
# == Retrieval crawlers: blocking these can remove you from AI answers ==
# Perplexity says Perplexity-User generally ignores robots.txt. OpenAI says
# robots.txt rules may not apply to ChatGPT-User.
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: YouBot
Disallow: /
User-agent: meta-webindexer
Disallow: /
User-agent: Amzn-SearchBot
Disallow: /
User-agent: DuckAssistBot
Disallow: /
User-agent: MistralAI-Index
Disallow: /
# == Your Standard Rules ==
User-agent: *
Disallow: /admin/
Disallow: /private/
# == Sitemap Location ==
Sitemap: https://www.your-website.com/sitemap.xmlTemplate 2: Allow All Known AI Crawlers
Use this template if your goal is to maximise your visibility and ensure your content is discoverable by AI search and answer engines.
# === ALLOW ALL MAJOR AI CRAWLERS ===
# Template provided by paceghost.io - The AI Readiness Platform
# Last updated: October 13, 2026
# This template ensures AI crawlers can access your site while still protecting
# sensitive areas (like admin panels). By default, crawlers are allowed,
# so we only need to specify what to disallow.
# == Default Rules for All Crawlers (including AI) ==
User-agent: *
Disallow: /admin/
Disallow: /dashboard/
Disallow: /login/
# == Sitemap Location ==
Sitemap: https://www.your-website.com/sitemap.xmlTemplate 3: Block Training, Keep AI Search
The position most site owners actually want, and the one the first two templates cannot express. Your content stays out of foundation-model training while remaining quotable and linkable in ChatGPT, Claude and Perplexity answers.
# === BLOCK AI TRAINING, KEEP AI SEARCH ===
# Template provided by paceghost.io - The AI Readiness Platform
# Last updated: October 13, 2026
#
# Use this if you do not want your content used to train foundation models,
# but you DO want to remain quotable and linkable inside AI answers.
# == Training crawlers and opt-out tokens: blocked ==
# Google-Extended also controls grounding in Gemini Apps and Vertex AI. Delete
# its two lines if you want Gemini to use your content for grounding.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: MistralAI-Training
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# == Datasets and crawlers without an operator page: blocked ==
# Common Crawl publishes an open dataset that anyone can use. ByteDance and
# Cohere publish no page that says what these crawlers do. Delete these
# lines if you want to allow them.
User-agent: CCBot
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: Bytespider
Disallow: /
# == Retrieval crawlers: allowed on purpose, except private paths ==
# Their operators document them for search results and answers.
# Keep these paths the same as in your standard rules below.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: YouBot
User-agent: meta-webindexer
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
Disallow: /admin/
Disallow: /private/
# == Your Standard Rules ==
User-agent: *
Disallow: /admin/
Disallow: /private/
# == Sitemap Location ==
Sitemap: https://www.your-website.com/sitemap.xmlWhat's Next?
Defining your rules is a critical first step. But the AI landscape changes daily. New crawlers appear constantly, and you're left wondering:
- Is my robots.txt file still effective?
- Are bots respecting my rules?
- How is AI actually interpreting my content?
That's where Paceghost comes in. We're building the toolkit to turn this manual checklist into an automated, effortless dashboard.
Related reading
Stay in the loop
The AI crawler landscape moves fast.
We update this list as new bots emerge. Subscribe to get notified when we publish new resources, templates, and guides on AI readiness.