ClaudeBot
ClaudeBot is Anthropic's web crawler, identified by the user-agent string ClaudeBot, that fetches public web pages for Claude's AI models, and site owners can allow or block it through a rule in robots.txt like any other named bot.
ClaudeBot is the crawler Anthropic runs to collect public web content for its Claude models. It identifies itself with the user-agent ClaudeBot, honors robots.txt rules written for that name, and publishes a contact address in its user-agent string. Anthropic also documents separate agents for user-triggered fetches and for search-related retrieval, following the same pattern OpenAI uses with GPTBot and OAI-SearchBot.
Compared with the other AI crawlers, ClaudeBot gets less attention from small businesses, and the reason is honest: Claude is a smaller referral source. Within AI referral traffic, Claude accounts for about 2.62% of visits, versus 74.78% for ChatGPT, according to Similarweb-family data. Claude also has no Maps integration and no dedicated local-business surface the way Google does, so it's not where most homeowners go to find a roofer.
That said, nobody has measured how Claude recommends local businesses. There's no equivalent of the SOCi or Ahrefs studies for Claude specifically, and any percentage you see claiming to describe Claude's local citation behavior is not backed by published data. Speak qualitatively or not at all. What's known is that when Claude searches the web to answer a question, it reads pages the same way any engine does, so the basics apply: clear facts, readable pages, consistent listings.
For a dental practice or HVAC company, the decision on ClaudeBot is low stakes either way. Allowing it costs nothing and keeps Claude able to read your site when a user asks it to search. Blocking it is a defensible choice if you'd rather your content not be used by Anthropic. Just make sure the rule is deliberate rather than a leftover from a blanket AI-bot block.
Tibly tracks Claude as one of its seven engines, so you can see whether Claude names your business at all for the buying questions in your city, and its AI-readiness audit checks whether ClaudeBot is allowed by your robots.txt.
Keep exploring the glossary
- AI CrawlerAn AI crawler is an automated bot such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, or Google-Extended that fetches web pages so an AI company can train its models or retrieve current information to answer user questions.
- GPTBotGPTBot is OpenAI's web crawler for collecting publicly available content to train its AI models, identified by the user-agent string GPTBot and controllable through robots.txt, and it is separate from OAI-SearchBot, which powers ChatGPT's search answers.
- PerplexityBotPerplexityBot is the web crawler Perplexity uses to index pages for its AI answer engine, identified by the user-agent PerplexityBot and controllable through robots.txt, and it feeds an engine that runs a fresh web search for every question.
- robots.txt for AI Crawlersrobots.txt for AI crawlers means using the robots.txt file at your site's root to allow or block named AI bots such as GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot, deciding which AI systems may read your pages for training or live answers.
Common questions about ClaudeBot
- Does Claude recommend local businesses?
- Sometimes, when a user asks it to search. But nobody has published a measurement of how often or by what pattern, and Claude has no Maps or local-business integration. Treat any specific number about Claude's local recommendations as unverified. Tibly tracks Claude directly so you can see your own results.
- How do I block ClaudeBot?
- Add a robots.txt rule with User-agent: ClaudeBot and Disallow: /. Anthropic states ClaudeBot respects these rules, and the change takes effect on its next visit. Before you do, confirm you're not also blocking crawlers from engines that send you more traffic, like OAI-SearchBot or PerplexityBot.
- Is ClaudeBot a training crawler or a search crawler?
- ClaudeBot is Anthropic's general crawler for collecting public web content, closer to GPTBot than to OAI-SearchBot. Anthropic documents separate agents for user-triggered fetches and search retrieval. If you want Claude to be able to read your site when a user asks it to search, leave those agents allowed.