Why would AI engines fail to crawl my site?

Common causes include robots.txt rules that block AI user-agents, server errors, redirect loops, login walls, or content that only appears after JavaScript runs. The checker surfaces each of these so you can fix the right one.

Does JavaScript rendering affect AI visibility?

Often, yes. Some AI crawlers do not fully execute JavaScript, so content injected client-side can be invisible to them. The checker flags pages where your important text depends on JS to load.

Will blocking AI bots hurt my visibility?

If you block GPTBot or similar crawlers in robots.txt, those engines cannot cite you at all. The checker shows which bots you are currently allowing or blocking so the choice is deliberate, not accidental.

How do I check if GPTBot is blocked on my site?

Fetch your robots.txt file and search for "GPTBot". If it appears under a Disallow directive or if you have a wildcard "User-agent: *" with "Disallow: /" and no explicit Allow for GPTBot, OpenAI's crawler is blocked. You can also run a curl request with the GPTBot user-agent string to test your server response directly.

Can I allow some AI bots but block others?

Yes. robots.txt supports per-user-agent rules, so you can write separate Allow/Disallow directives for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended independently. This lets you, for example, allow Perplexity for real-time search while blocking OpenAI training crawls.

What is the difference between crawlability and indexability?

Crawlability is whether a bot can reach and fetch the page. Indexability is whether the bot can store and use the content once fetched. A page can be crawlable but not indexable (e.g., blocked by noindex) or indexable in principle but unreachable because of robots.txt blocks or server errors.

Does allowing AI bots affect my server load?

AI crawlers are generally well-behaved and respect crawl-delay directives. GPTBot and ClaudeBot follow robots.txt and do not crawl at rates that stress typical servers. If you are concerned, set a Crawl-delay directive in robots.txt to rate-limit AI bot requests.

Does my CDN or WAF block AI crawlers?

Possibly. Some CDN and Web Application Firewall configurations use bot-management rules that block user-agents not on a known-good list. Since AI bots are relatively new, older bot-management configurations may reject them. Check your WAF/CDN bot rules if you suspect blocking.

What is llms.txt and should I add it?

llms.txt is a file placed at your domain root that provides structured guidance to AI language models about your site's content. It is not yet a universal standard, but major AI developers have signaled support for it. Adding it is low-effort and positions your site well as the standard matures.

How does GEO crawlability relate to SEO crawlability?

They overlap significantly — both care about robots.txt, server responses, and redirect hygiene — but GEO crawlability adds AI-specific concerns: which user-agents are permitted, JavaScript rendering behavior for non-rendering crawlers, and the llms.txt signal. A technically healthy SEO site may still fail GEO crawlability checks.

Will blocking AI crawlers protect my content from being used in AI training?

Blocking GPTBot prevents OpenAI from crawling new content for training after you add the block. Content already in training data before the block is not retroactively removed. Google-Extended specifically controls Gemini/Bard training. Separate blocks are needed for each AI provider.

Free Tool

Free GEO Crawlability Checker

Find out whether AI engines can actually crawl and render your pages — before they decide whether to read, trust, and cite you.

What is GEO crawlability?

Generative Engine Optimization (GEO) starts with one question: can the AI even reach your content? Engines like GPTBot, ClaudeBot, and PerplexityBot have to fetch and render a page before any of it can appear in an answer. If they are blocked or the content only loads via JavaScript, you are invisible.

The GEO Crawlability Checker tests your robots.txt rules, AI bot access, server responses, and whether your key content renders without JavaScript. It tells you, in plain terms, whether the engines that power AI answers can read your site at all.

How it works

Paste your URL

Enter any page and we simulate how AI crawlers request and load it.

We test bot access

We check robots.txt, AI user-agent rules, status codes, and JavaScript-dependent content.

Get your fix list

See exactly what blocks AI engines and how to open your content up to them.

Frequently asked questions

The Complete Guide to GEO Crawlability: Can AI Engines Read Your Site?

Generative Engine Optimization (GEO) begins at the most fundamental level: can the AI even see your content? Before any model can decide whether to trust, summarize, or cite your pages, a crawler must successfully fetch and render them. AI engines dispatch specialized bots — GPTBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, and Google-Extended for Gemini — each with its own user-agent string. If your server or robots.txt turns any of them away, that engine cannot include your content in its answers, regardless of how well written it is.

The challenge is that most websites were configured for search engine crawlers, not AI bots. A robots.txt file that was last edited in 2019 almost certainly does not account for GPTBot (introduced in 2023) or ClaudeBot. Without an explicit Allow directive, some server configurations default to blocking unknown agents. The result is silent: your pages appear accessible in a browser, your SEO rankings look fine, but AI engines receive a 403 or are turned away by robots.txt — and you never know unless you specifically test for it.

JavaScript-rendered content is a second major GEO crawlability barrier. Many modern frameworks (React, Vue, Angular) deliver a nearly empty HTML skeleton and load the actual content client-side after JavaScript executes. Traditional Googlebot runs a two-stage rendering pipeline that handles this; many AI bots do not. If your main navigation, product descriptions, blog content, or FAQ answers only appear after JS runs, those elements are invisible to crawlers that do not execute JavaScript. Server-side rendering (SSR) or static generation eliminates this class of problem entirely.

Redirect chains, soft 404s, and login walls round out the most common GEO crawlability failures. A chain of three or more redirects before reaching the final page can cause a crawler to give up before retrieving any content. Pages that return 200 OK but display "content not found" or a sign-in form (soft 404s) waste crawl budget and return no usable content. GEO crawlability testing must simulate AI user-agents specifically — using a standard SEO crawler with Googlebot settings will miss all of these AI-specific access failures.

GEO Crawlability Fix Playbook

Eight targeted steps to ensure every major AI engine can reach and read your pages.

Audit robots.txt for AI user-agents

Open your robots.txt and search for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. If these agents are explicitly Disallowed or absent from an Allow rule on a wildcard-blocked site, AI engines cannot crawl you. Add explicit Allow rules for each AI bot you want to permit.

Test server responses with AI user-agent headers

Use curl or a similar tool to request your pages with each AI user-agent string. Look for 403, 429, or redirect loops that only appear for specific agents. Some WAF configurations and bot-protection services block unfamiliar user-agents by default — AI bots included.

Identify JavaScript-only content blocks

Disable JavaScript in your browser and visit each key page. Any content that disappears — navigation, product listings, article bodies, FAQ sections — is invisible to AI crawlers that do not execute JS. Prioritize moving this content to server-rendered or statically generated HTML.

Add or update your llms.txt file

llms.txt is an emerging standard that signals to AI crawlers which pages are authoritative, which can be included in training data, and which contain the most valuable content. Place it at your root (/llms.txt) and update it as your content evolves.

Audit redirect chains on important pages

Use a redirect checker to trace the full chain for your most important URLs. Any chain longer than two hops should be collapsed to a single direct redirect. Pay special attention to HTTP-to-HTTPS and www-to-non-www chains that are often overlooked.

Ensure key pages return 200 OK

Run a full crawl and flag any page that returns a non-200 status code or that returns 200 but delivers no meaningful content. Soft 404s — pages that return 200 but say "page not found" or require login — are especially costly because they consume crawl budget without delivering any content.

Test with multiple AI user-agents

Do not assume that allowing GPTBot is sufficient. Each AI engine has its own bot with its own user-agent, and your server configuration may block some while allowing others. Test GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and Applebot-Extended individually.

Monitor crawlability with a GEO checker

GEO crawlability is not a static property — it changes when you update robots.txt, deploy new framework changes, or add a CDN or WAF. Run a GEO crawlability check after every significant technical deployment to catch regressions before they affect AI visibility.

GEO Crawlability Glossary

Plain-language definitions for every technical term in AI engine crawlability.

GPTBot: OpenAI's web crawler, used to gather information for ChatGPT and related models. Identified by the user-agent string "GPTBot". Can be allowed or blocked in robots.txt with a "User-agent: GPTBot" directive.
ClaudeBot: Anthropic's web crawler, used to gather information for Claude models. Identified by the user-agent string "ClaudeBot". Must be explicitly allowed in robots.txt to permit Anthropic to crawl your site.
PerplexityBot: Perplexity AI's web crawler, used for real-time web search that powers Perplexity answers. Unlike some AI bots, PerplexityBot actively crawls pages during live searches, making it particularly important for real-time AI visibility.
Google-Extended: Google's opt-out mechanism for AI training. Setting "User-agent: Google-Extended / Disallow: /" prevents Google from using your content to train Gemini and Bard models, while still allowing Googlebot for search indexing.
GEO (Generative Engine Optimization): The practice of making content accessible, parseable, and trustworthy for AI-powered generative engines that synthesize answers from web content. GEO extends traditional SEO with a focus on AI crawler access and content extractability.
robots.txt: A text file at the root of a domain that instructs crawlers which pages or paths they are allowed or forbidden to access. AI engines check robots.txt before crawling; violations of Disallow directives are honored by reputable bots.
User-Agent: A string sent in HTTP request headers that identifies the browser, bot, or crawler making the request. Servers and robots.txt files use user-agent strings to distinguish between Googlebot, GPTBot, human browsers, and other agents.
JavaScript Rendering: The process of executing JavaScript on a page to generate its final HTML. Pages that require JS rendering may be partially or fully invisible to AI crawlers that do not execute JavaScript before reading content.
Server-Side Rendering (SSR): A web architecture where the full HTML of a page is generated on the server before delivery to the browser. SSR content is immediately readable by any crawler, including AI bots that do not execute JavaScript.
llms.txt: An emerging standard file (placed at /llms.txt) that provides AI language models with structured guidance about a site's content: which pages are most important, which are suitable for inclusion, and key facts about the organization.
Crawl Budget: The number of pages a crawler will fetch from a site within a given time period. Wasted crawl budget on redirect chains, soft 404s, and blocked URLs reduces how much valuable content a crawler indexes.
Disallow Directive: A robots.txt instruction that forbids a named user-agent from accessing a specified path. Example: "User-agent: GPTBot / Disallow: /" blocks GPTBot from crawling the entire site.

Who Needs a GEO Crawlability Checker

Sites and scenarios where AI crawlability failures are most likely and most costly.

SPA and React/Next.js Sites

Single-page applications that rely heavily on client-side JavaScript rendering are at highest risk of invisible content for AI crawlers. Any page where content loads via API calls after the initial HTML delivery needs careful SSR or static generation review.

Enterprises with Complex robots.txt

Large enterprise sites often have robots.txt files maintained by multiple teams over years, resulting in conflicting rules and unintended blocks. A single misplaced Disallow rule can silently exclude entire sections of a site from AI crawling.

Publishers Controlling AI Training

Publishers who want control over whether their content is used for AI training need to know exactly which bots are crawling them. The GEO Crawlability Checker shows the current access status for every major AI bot, making deliberate opt-in or opt-out decisions straightforward.

E-commerce with JS-loaded Products

Product detail pages that load prices, availability, and descriptions via JavaScript are common in e-commerce. If those elements are invisible to AI crawlers, the products cannot appear in AI shopping answers or brand citations.

Agencies Auditing Client Sites

Agencies launching or migrating client sites need to verify GEO crawlability as part of pre-launch QA. A crawlability check catches access issues before they silently degrade AI visibility for weeks or months after launch.

New Sites Setting AI Policy

Organizations building new sites have the opportunity to set their AI access policy from day one: which bots to allow for which sections, whether to publish an llms.txt, and how to structure content for maximum AI extractability.