GPTBot and other AI crawlers: which ones to let into your site
GPTBot isn't OpenAI's only bot: it has 4 and each does something different. Here's which one decides whether you show up in ChatGPT search, which one trains models and how to check your site in 15 minutes.
By Rubén García · November 13, 2026 · 5 min
Your website may be shutting ChatGPT out without you knowing. Since July 2025, Cloudflare has asked every new site whether to let AI crawlers (programs that read web pages) in. Cloudflare is a service that protects and speeds up 1 in 5 websites worldwide. And those programs don't all do the same thing.
4
OpenAI bots with different jobs: search, model training, visits a user asks for and ad review (OpenAI)
20%
of the web runs through Cloudflare, which since July 2025 asks every new site whether to let AI in (Cloudflare)
24 h
is how long ChatGPT search may take to apply a change to your robots.txt, the file that says who can read your site (OpenAI)
What is GPTBot and how is it different from OpenAI's other bots?
GPTBot is the program OpenAI uses to read websites and train its AI models. Blocking it tells OpenAI not to use your content for training. But it's not the one that decides whether you appear in ChatGPT.
- OAI-SearchBot: the one behind ChatGPT search. If you block it, according to OpenAI your site won't appear in its answers, although it can still show up as a navigational link.
- GPTBot: reads websites to train models. Blocking it doesn't remove you from search.
- ChatGPT-User: visits when a person asks ChatGPT to read a page. OpenAI warns that robots.txt rules may not apply.
- OAI-AdsBot: only visits pages you submit as ads on ChatGPT, to review them.
In practice, ‘blocking OpenAI’ isn't one decision, it's four. OpenAI says each setting is independent of the others. So decide separately whether you want to appear in its search and whether you mind your content being used for training.
Search and training are two different doors.
Which AI crawlers should you let into your site and which should you block?
It depends on what you want. This guide covers the most common cases for a small business:
- You want ChatGPT to cite you in its answers: let OAI-SearchBot in.
- You don't want your content to train OpenAI's models: block GPTBot. You can still appear in ChatGPT search.
- You don't want Google to use your content to train Gemini: block Google-Extended. According to Google, it doesn't affect your ranking in its search engine.
- Never block Googlebot, Google Search's crawler, unless you want to disappear from Google.
At Garcai we think that, for a small business that lives off being found, shutting out ChatGPT search is rarely worth it. Training is a matter of principle, and each brand decides. If you want to know how to earn those citations, see our guide to showing up in ChatGPT.
Could Cloudflare be blocking ChatGPT on your site without you knowing?
Your take, in one click
Do you let AI read your website?
It can happen. On 1 July 2025, Cloudflare started asking every new site whether to let AI crawlers in. Before that, more than a million customers had already switched on its one-click block.
If someone switched that block on when your site was set up, perhaps no AI bot is getting in. And you won't see it in robots.txt, because it works separately. Ask whoever runs your site whether you use Cloudflare and what's switched on.
The Garcai method: check which AI crawlers get into your site in 15 minutes
You don't need to code for the first three steps.
- Open your robots.txt: type your domain followed by ‘/robots.txt’ into your browser. If it doesn't exist, every program can read your site.
- Look for the names GPTBot, OAI-SearchBot and Google-Extended. If ‘Disallow’ with a slash appears next to one, that program is blocked.
- Decide using the guide above: do you want to appear in ChatGPT search? Do you mind your content training models?
- Ask for the change, and ask them to check the AI bot block in Cloudflare or your hosting too.
- A month later, check in Google Analytics whether visits arrive from chatgpt.com.
You can do it yourself: steps 1, 2, 3 and 5.
Ask your tech team or whoever runs your site: step 4. Message ready to copy: ‘We want to appear in ChatGPT search. Can you allow OAI-SearchBot in robots.txt and check whether Cloudflare blocks AI bots?’
If you're worried that what AI says about you is wrong, also check the AI Overviews for brand searches on Google.
Common mistakes when blocking GPTBot and other AI crawlers
- Blocking every AI bot at once and vanishing from ChatGPT search by accident.
- Believing that blocking GPTBot removes you from ChatGPT: it only affects model training.
- Thinking robots.txt is a lock: Google explains it's a request and some programs ignore it.
- Only checking robots.txt and forgetting the Cloudflare or hosting block.
- Blocking Googlebot thinking it's an AI bot, and dropping out of Google.
Blocking ‘AI’ without looking at what each program does is shutting the door blindfolded. Let in the one that cites you; decide about the one that learns. And if you're getting started with SEO, see our guide to SEO tools for small businesses.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT search?
No. GPTBot is for training models; what counts for ChatGPT search is OAI-SearchBot.
What happens if I block OAI-SearchBot?
According to OpenAI, your site won't appear in ChatGPT search answers. It can still show up as a navigational link.
Does blocking Google-Extended affect my Google ranking?
No. Google says it doesn't affect whether your site appears in its search engine or how it ranks.
Does robots.txt stop AI from reading my site?
Not entirely: it's a request that reputable programs respect, but others don't. To truly hide something, protect it with a password.
