How to Allow GPTBot in robots.txt (and Verify It Works)
To allow GPTBot, add a User-agent: GPTBot
block with Allow: / to your robots.txt and
make sure no broader rule blocks it. Then verify the bot can actually
reach your pages with a live crawl probe, a rule that
looks right can still be undone by a conflicting wildcard or a
firewall.
GPTBot is OpenAI's web crawler. It gathers content that helps ChatGPT discover, understand and cite websites. If you want to appear in ChatGPT's answers, GPTBot needs to be allowed. Blocking it, often by accident, makes you invisible to one of the largest AI audiences on the web.
The robots.txt rules to add
robots.txt lives at the root of your domain
(https://yoursite.com/robots.txt). To explicitly welcome
the major AI crawlers, add:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Here's what each one controls:
- GPTBot, OpenAI's crawler for training and improving ChatGPT.
- OAI-SearchBot, OpenAI's crawler for ChatGPT search results and citations.
- ClaudeBot, Anthropic's crawler for Claude.
- PerplexityBot, Perplexity's answer-engine crawler.
- Google-Extended, Google's control for Gemini and AI Overviews.
Watch out for the rule that silently blocks everything
The most common reason AI crawlers get locked out isn't an explicit block, it's a broad catch-all. If your file contains something like:
User-agent: *
Disallow: /
…then every bot without its own named block is blocked. Crawlers match
the most specific user-agent block, so adding the named
Allow rules above protects the AI bots even if a wildcard
disallow exists. Double-check there's no stray Disallow: /
inside a GPTBot block left over from a past "block AI" decision.
Allowing AI crawlers ≠ giving up control
You can still protect private paths. Keep AI bots out of admin, checkout or staging areas while allowing the content you want cited:
User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Verify it actually works
A correct-looking rule can still fail in practice, a CDN, WAF or bot-management firewall may block the crawler's IP before robots.txt even matters. So don't assume; verify. Sona's free AI visibility checker runs a live GPTBot probe that confirms whether OpenAI can actually fetch your pages right now, alongside 16 other AI-readiness checks.
Allowing crawlers is step one. To understand the full picture of what makes AI engines cite you, read how to check your website's AI visibility and how to get cited by ChatGPT and Perplexity.
Is GPTBot actually reaching your site?
Run a free live crawl probe and confirm AI engines can read your pages.
Check your site free →