FREE TOOL

Are AI Crawlers Allowed On Your Site?

Check in one click whether GPTBot, ClaudeBot, Google-Extended and other AI crawlers can read your pages — or whether your robots.txt is quietly blocking them.

What is an AI crawler?

AI crawlers are bots run by companies like OpenAI, Anthropic and Google to read public web pages and feed their AI models and answer engines. If they can read your site, your brand can show up in ChatGPT, Claude, Perplexity and AI Overviews.

Your robots.txt file — a plain text file at the root of your domain — tells these bots what they may and may not access. A single Disallow line can quietly remove you from AI answers.

How it works

Enter your domain

Type your website address. We fetch your public /robots.txt file.

We parse the rules

We match each major AI crawler against your User-agent and Disallow rules.

See who is blocked

Get a clear allowed / blocked status for every AI crawler in seconds.

How robots.txt controls AI crawlers in 2026

Your robots.txt file is a plain-text file at the root of your domain that tells automated bots which parts of your site they may and may not fetch. For two decades it governed search engine spiders like Googlebot. Today it has a second, equally important job: deciding which AI crawlers — the bots that feed ChatGPT, Perplexity, Claude and Google’s generative answers — are allowed to read your content.

AI vendors run distinct user-agents for distinct purposes. OpenAI uses GPTBot to gather training data, OAI-SearchBot to power live search answers in ChatGPT, and ChatGPT-User for on-demand fetches when a user asks ChatGPT to open a link. Anthropic runs ClaudeBot and the older anthropic-ai. Perplexity uses PerplexityBot. Google uses Google-Extended as an opt-out token for Gemini training, separate from its Googlebot search crawler. Common Crawl’s CCBot feeds many downstream models, and Bytespider (ByteDance), Amazonbot and Applebot-Extended round out the list.

The key insight is that these bots do different things, so a blanket block is rarely the right move. Blocking a training crawler protects your content from being absorbed into a model, but blocking a live-retrieval bot like OAI-SearchBot or PerplexityBot can quietly remove you from the AI answers your buyers see — the AI equivalent of deindexing yourself from Google. Most brands want the opposite: maximum visibility in AI answers, with selective control over bulk scraping.

robots.txt is a directive, not a hard wall. Reputable crawlers from OpenAI, Anthropic, Perplexity and Google honor it, but it relies on the bot choosing to obey. It is the standard, low-friction way to express your preferences, and pairing it with an llms.txt file gives well-behaved AI systems a clean map of the content you most want them to use.

How to control AI crawlers with robots.txt

A practical sequence for auditing what you allow today, opening the doors that grow AI visibility, and closing the ones you do not want — without accidentally hiding from ChatGPT and Perplexity.

Audit your current rules

Run your domain through this checker to see which AI user-agents you allow, block or partially block today. Many sites unknowingly block GPTBot or PerplexityBot because of a copied-and-pasted rule or a security plugin default.

Decide your allow-vs-block strategy

Separate live-retrieval bots (OAI-SearchBot, ChatGPT-User, PerplexityBot) from training bots (GPTBot, Google-Extended, CCBot, ClaudeBot). Most brands allow retrieval bots for visibility and make a deliberate choice on training bots.

Allow the AI crawlers that matter

To appear in ChatGPT and Perplexity answers, explicitly allow OAI-SearchBot, ChatGPT-User and PerplexityBot. If you want your content used in answers and training, allow GPTBot and ClaudeBot too.

Block the scrapers you do not want

If bandwidth, bulk scraping or unattributed reuse is a concern, add Disallow rules for aggressive crawlers such as Bytespider, CCBot or Amazonbot while leaving the answer-engine bots allowed.

Write correct, specific syntax

Use one User-agent block per bot, name the user-agent verbatim, and use Allow: / or Disallow: / under it. A trailing rule that is too broad can block far more than you intend, so keep each directive scoped.

Test before and after you publish

Re-run this checker after editing robots.txt to confirm each bot reads as you intended. Verify the file returns HTTP 200 at /robots.txt and is not blocked by a firewall, CDN or WAF rule.

Add an llms.txt to guide good bots

Publish an llms.txt that lists your most important pages and documentation. It complements robots.txt by pointing well-behaved AI systems at the content you most want cited.

Monitor crawler activity over time

AI vendors add and rename user-agents regularly. Re-check quarterly, watch your server logs for new AI bots, and confirm a CDN update or new plugin has not silently changed your rules.

AI crawler and robots.txt glossary

The vocabulary of controlling AI bots, in plain English.

robots.txt: A plain-text file at your domain root (yoursite.com/robots.txt) that tells crawlers which paths they may and may not fetch. The standard way to express crawl preferences to search and AI bots.
User-agent: The identifier a bot sends to announce who it is. In robots.txt, each User-agent line names the bot a rule block applies to, such as GPTBot or PerplexityBot.
Disallow: A directive that asks a named user-agent not to fetch the listed path. Disallow: / blocks the whole site for that bot; Disallow: /private/ blocks only that folder.
Allow: A directive that explicitly permits a path, used to carve exceptions inside a broader Disallow or to clearly grant access to a specific bot.
Crawl-delay: An optional directive requesting a minimum number of seconds between requests from a bot, used to limit server load. Honored by some crawlers but not all AI bots.
GPTBot: OpenAI’s crawler that gathers web content used to train its models. Blocking GPTBot opts your content out of training but does not by itself remove you from live ChatGPT search answers.
Google-Extended: A robots.txt token Google uses to let you opt out of having your content used to train and ground Gemini, separate from the Googlebot crawler that powers Search. Blocking it does not affect your Google Search rankings.
llms.txt: A proposed plain-text file that lists your key pages and documentation in a clean, machine-readable form so AI systems can find your most important content. Complements rather than replaces robots.txt.
AI crawler: A bot operated by an AI vendor to fetch web pages for training, live retrieval, or on-demand browsing. Examples include GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and CCBot.
Live retrieval bot: An AI crawler that fetches pages at answer time to ground a response, such as OAI-SearchBot, ChatGPT-User and PerplexityBot. Blocking these can remove you from AI answers.
Training crawler: A bot that collects content to train a model rather than to answer a live query, such as GPTBot, Google-Extended, CCBot and Bytespider.

Who uses the Robots.txt Checker

If buyers research you through AI assistants, your robots.txt now decides whether they can read you.

Publishers and media

Decide deliberately whether to allow training crawlers like GPTBot and CCBot while keeping answer-engine bots such as OAI-SearchBot and PerplexityBot allowed so your reporting still gets cited.

SaaS and software

Confirm ChatGPT and Perplexity can read your docs and feature pages, so when buyers ask AI for the best tool in your category, your site is available to be quoted.

Ecommerce and DTC

Make sure product and category pages are crawlable by AI shopping answers, and selectively limit aggressive scrapers that strain your servers without sending traffic.

Agencies and consultants

Audit a client’s robots.txt in seconds, spot accidental blocks of GPTBot or PerplexityBot, and turn the fix into a clear, measurable AI-visibility win.

Frequently asked questions

What is GPTBot?

GPTBot is OpenAI’s web crawler. It reads public pages to help train and ground its models. Blocking GPTBot in robots.txt stops OpenAI from using your content.

Should I allow AI crawlers?

For most brands, yes. Allowing AI crawlers lets your content appear in ChatGPT, Claude and Perplexity answers — a fast-growing source of discovery and traffic.

Does blocking robots.txt remove me from ChatGPT?

Blocking ChatGPT-User and GPTBot stops new crawling and live retrieval, which reduces how often you are cited. It does not erase content already learned, but it limits future visibility.

What is Google-Extended?

Google-Extended is a robots.txt token that controls whether Google can use your content for Gemini and AI features, separately from normal Google Search indexing.

How is this different from Rankfender?

This tool checks access. Rankfender goes further — measuring whether AI engines actually mention your brand and showing how to win those citations.

Should I block AI crawlers?

For most brands, no — at least not the live-retrieval bots. Blocking OAI-SearchBot, ChatGPT-User or PerplexityBot can remove you from the answers buyers see in ChatGPT and Perplexity. The nuanced choice is around training crawlers like GPTBot, Google-Extended and CCBot: block those if you do not want your content absorbed into models, but understand that blocking everything trades away AI visibility to protect content you may want discovered anyway.

Does blocking GPTBot remove me from ChatGPT?

Not directly. GPTBot is OpenAI’s training crawler, not the bot that fetches pages for live answers. ChatGPT’s search answers are powered by OAI-SearchBot, and on-demand link opening by ChatGPT-User. If you want to stay out of training but remain visible in ChatGPT answers, block GPTBot while allowing OAI-SearchBot and ChatGPT-User.

What is Google-Extended?

Google-Extended is a robots.txt user-agent token that lets you opt out of having your content used to train and ground Google’s Gemini models. It is separate from Googlebot. Blocking Google-Extended does not affect your Google Search rankings or whether you appear in normal search — it only governs generative AI training and grounding.

Is allowing AI bots bad for SEO?

No. Allowing AI crawlers does not harm your classic Google rankings; Googlebot and AI training tokens like Google-Extended are governed separately. Allowing answer-engine bots actually helps you appear in ChatGPT and Perplexity, which is increasingly part of how buyers discover brands. There is no SEO penalty for letting AI crawlers read your public content.

How do I block an AI crawler in robots.txt?

Add a block naming the user-agent, then a Disallow rule. For example: User-agent: GPTBot then Disallow: / blocks GPTBot from the whole site. Repeat one block per bot you want to restrict. After editing, re-run this checker and confirm /robots.txt returns HTTP 200 so the rule is actually being served.

What is the difference between robots.txt and llms.txt?

robots.txt controls which bots may fetch which paths — it is a permission layer. llms.txt is a proposed file that lists your most important pages and documentation so AI systems can find your best content quickly. They work together: robots.txt sets access, llms.txt guides well-behaved AI to what you most want cited.

Will AI crawlers obey my robots.txt?

Reputable crawlers from OpenAI, Anthropic, Perplexity and Google publicly commit to honoring robots.txt. It is a directive, not a hard block, so a malicious scraper can ignore it — but the major AI vendors that drive ChatGPT, Claude, Perplexity and Gemini answers do respect it, which makes robots.txt the practical control point for AI visibility.

How often should I check my robots.txt for AI bots?

At least quarterly, and after any CDN, security plugin or platform change. AI vendors regularly add or rename user-agents — Applebot-Extended and OAI-SearchBot are recent examples — so a file that was correct last year may now miss new bots or carry an outdated block. Re-running this checker takes seconds and catches silent changes.

Allowed isn’t the same as mentioned

Access is step one. See whether AI engines actually recommend your brand — and how to make them.

Get My Free RAIVE Score