AI Crawler
An AI crawler is an automated bot such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, or Google-Extended that fetches web pages so an AI company can train its models or retrieve current information to answer user questions.
Also known as: AI bot, LLM crawler
An AI crawler is a bot that visits your website on behalf of an AI company. It works like Googlebot, requesting pages and reading their contents, but the purpose is different. Some AI crawlers collect text to train future models. Others fetch pages in real time so an answer engine can ground its response in current information. A few do both, under different names, from the same company.
The main ones you'll see in server logs: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot from Perplexity, and the Google-Extended token that Google uses to control Gemini's access to your content. Each identifies itself with a user-agent string, and each respects robots.txt rules aimed at that name, according to the companies' own documentation. That's what lets you allow one and block another.
The distinction between training crawlers and search crawlers is the one that matters for your business. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the one that powers ChatGPT's search features. Block GPTBot and you've opted out of future training data. Block OAI-SearchBot and ChatGPT can't fetch your pages when someone asks for a roofer in your town. Many site owners blocked everything in 2023 and never revisited the decision.
For a plumber or dental office, the question is simple: do you want AI answer engines to be able to read your prices, hours, and service list? If yes, the search-oriented crawlers need access. A blanket block on all AI bots, which some hosting providers and security plugins apply by default, quietly removes you from the sources an engine can cite. Nobody has measured exactly how much visibility that costs, but a source that can't be read can't be named.
Tibly's AI-readiness audit checks whether the major AI crawlers can reach your site and flags a robots.txt or firewall rule that shuts them out. On a Tibly-hosted site, crawler access is open by default, and new or changed pages are pushed to search indexes via IndexNow so they're available to engines quickly.
Keep exploring the glossary
- GPTBotGPTBot is OpenAI's web crawler for collecting publicly available content to train its AI models, identified by the user-agent string GPTBot and controllable through robots.txt, and it is separate from OAI-SearchBot, which powers ChatGPT's search answers.
- OAI-SearchBotOAI-SearchBot is the OpenAI crawler that indexes web pages for ChatGPT's search features, so a site that blocks it can't be fetched or linked when ChatGPT searches the web, and it is separate from GPTBot, OpenAI's training crawler.
- ClaudeBotClaudeBot is Anthropic's web crawler, identified by the user-agent string ClaudeBot, that fetches public web pages for Claude's AI models, and site owners can allow or block it through a rule in robots.txt like any other named bot.
- robots.txt for AI Crawlersrobots.txt for AI crawlers means using the robots.txt file at your site's root to allow or block named AI bots such as GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot, deciding which AI systems may read your pages for training or live answers.
Common questions about AI Crawler
- Should I block AI crawlers on my business website?
- Probably not the search ones. Blocking training crawlers like GPTBot is a reasonable choice about your content. Blocking OAI-SearchBot, PerplexityBot, and similar retrieval crawlers means answer engines can't read your pages, so they can't cite you when a customer asks who to hire.
- How do I know which AI crawlers visit my site?
- Check your server logs or hosting analytics for user-agent strings like GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot. Each company publishes its crawler names and, in most cases, IP ranges you can verify against. Tibly's AI-readiness audit also tests whether they can reach your pages.
- Do AI crawlers slow down my website?
- Rarely enough to notice on a small business site with a few dozen pages. If your logs show heavy bot traffic, check whether it's from a documented crawler with published IP ranges or from an unidentified scraper before you block anything, since a blanket block also removes you from AI answers.