robots.txt for AI Crawlers
robots.txt for AI crawlers means using the robots.txt file at your site's root to allow or block named AI bots such as GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot, deciding which AI systems may read your pages for training or live answers.
Also known as: AI bot blocking, Crawler directives for AI
robots.txt is the oldest access-control convention on the web: a text file at yourdomain.com/robots.txt listing which bots may fetch which paths. It was built for search crawlers and it now governs AI crawlers too, because every major AI company documents that its bots read and honor rules written for their names. The file doesn't enforce anything technically; it asks, and well-behaved crawlers comply.
The rules that matter for AI are per-bot. A User-agent: GPTBot block followed by Disallow: / tells OpenAI's training crawler to stay out. A separate User-agent: OAI-SearchBot block controls whether ChatGPT's search features can read you. ClaudeBot and PerplexityBot each take their own block. Google-Extended is a token rather than a bot: naming it controls whether Google may use your already-crawled content for Gemini training and grounding, and per Google's documentation it doesn't affect Search ranking or AI Overviews.
The mistake most small businesses inherit is the blanket block. When GPTBot was announced in 2023, many site owners, hosting providers, and security plugins added rules blocking every AI bot, or blocking all bots except Googlebot. If that rule is still in place, ChatGPT search, Perplexity, and Claude can't read your pages, and no amount of content work will get you cited by them. Some CDNs also ship an AI-bot block that operates at the network edge, outside robots.txt entirely, which the file won't show you.
A sensible default for a business that wants to be found: allow OAI-SearchBot, PerplexityBot, and the other retrieval crawlers, since they feed the answers customers read. Decide separately about training crawlers like GPTBot and ClaudeBot based on how you feel about your public content being used for model training; there's no measured visibility cost either way. Don't block Google-Extended unless you specifically want Gemini to stop using your content, since Gemini is the second-largest AI referral source at about 11.56% according to Similarweb-family data.
Tibly's AI-readiness audit reads your robots.txt for each named AI bot, reports which are allowed or blocked, and fetches your homepage as GPTBot and PerplexityBot to catch blocks that live in a CDN rather than the file. A Tibly-hosted site serves a robots.txt that allows all crawlers and points them to a sitemap, so the question never comes up.
Keep exploring the glossary
- AI CrawlerAn AI crawler is an automated bot such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, or Google-Extended that fetches web pages so an AI company can train its models or retrieve current information to answer user questions.
- GPTBotGPTBot is OpenAI's web crawler for collecting publicly available content to train its AI models, identified by the user-agent string GPTBot and controllable through robots.txt, and it is separate from OAI-SearchBot, which powers ChatGPT's search answers.
- OAI-SearchBotOAI-SearchBot is the OpenAI crawler that indexes web pages for ChatGPT's search features, so a site that blocks it can't be fetched or linked when ChatGPT searches the web, and it is separate from GPTBot, OpenAI's training crawler.
- Google-ExtendedGoogle-Extended is a robots.txt token that lets a site owner control whether Google uses their content for Gemini model training and grounding, without affecting how Googlebot crawls the site for Search or whether pages appear in AI Overviews.
Common questions about robots.txt for AI Crawlers
- Which AI bots should a small business allow in robots.txt?
- Allow the retrieval crawlers that power live answers: OAI-SearchBot, PerplexityBot, and Anthropic's search agent. Leave Googlebot and Bingbot allowed, since AI Overviews and Copilot depend on them. Training crawlers like GPTBot are a separate choice about your content, with no measured effect on whether you get recommended.
- My robots.txt allows everything but AI engines still can't read my site. Why?
- Probably a CDN or firewall rule. Several providers offer an AI-bot block that returns a 403 at the network edge before robots.txt is ever consulted. Check your CDN's bot settings, or fetch your homepage with an AI bot's user-agent string and see whether you get a page or an error.