Summarize with AI
Should you block GPTBot? For most early-stage startups, no. GPTBot only collects pages to train OpenAI's models, so blocking it does not take you out of ChatGPT search. The real risk is the copied block list that usually comes with it, because those lists also shut out the search crawlers that decide whether an assistant can name you when a buyer asks.
This matters more for a young company than for a famous one. You know what your product does. A buyer who asks an assistant for a tool in your category only gets what the assistant was able to read, and nobody is asking for you by name yet. At seed, being missing from that answer costs far more than letting a model learn from a few marketing pages.
Eleven Disallow lines under the comment # Block AI scrapers
Picture an example. A startup sells an AI scribe for small animal vet clinics. It listens during the exam, writes the SOAP note, and drafts the discharge sheet the pet owner takes home. In March, a thread about AI companies scraping websites goes round the founders' group chat. That afternoon the developer opens robots.txt and pastes in a block from a public gist.
The block starts with a comment, # Block AI scrapers. Under it sit eleven User-agent lines, each followed by Disallow: /. GPTBot is first. Further down are OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Google-Extended, plus three names nobody on the team looked up. The commit message says add AI bot blocks. Nobody reviews it, because it is a text file.
In June the founder opens ChatGPT and types the question his buyers type: what is the best AI scribe for a two-vet clinic? The answer names four products with one line each. Two of them launched after his did. He is not there. He asks Perplexity the same thing. Its Sources tab lists a rival's page titled AI scribe for veterinary practices, a comparison post on another rival's blog, and a practice management newsletter. Not one page from his site is cited.
The founder blames the content, and nobody opens the text file
He posts the screenshot in the team channel with a question mark. The developer replies that the robots.txt change only blocks training, because that is what the thread said. The thread was about GPTBot. The pasted list was about everything.
So the founder fixes what he can see. He asks the marketer for four new blog posts about AI in veterinary medicine. He adds a How did you hear about us field to the demo form. At a regional vet conference he hears practice owners describe their search in a way he had not expected, roughly like this as an illustration: I asked ChatGPT for a scribe and booked demos with the ones it named. He writes it down. He does not connect it to a file he has never opened.
The cost lands in the funnel before anyone can measure it. A buyer who asks an assistant has already named the problem and the type of clinic. She books calls with the names on her shortlist. The fourth vendor is not turned down. He is never considered, and his analytics record nothing, because the visit never happened. His ad spend now chases buyers who already hold a list.
GPTBot trains models. OAI-SearchBot puts you in answers.
Most advice treats AI crawlers as one group: block them or allow them. That is the mistake. Each vendor now runs separate bots for training and for search, and each one says so in its own documentation. Here is what each says, checked on the vendors' pages today.
OpenAI says GPTBot is used for training its foundation models, and that disallowing it signals your content should not be used for that. OAI-SearchBot is for search, and OpenAI states that sites opted out of it will not be shown in ChatGPT search answers. ChatGPT-User fetches pages when a person asks ChatGPT to, and OpenAI notes robots.txt rules may not apply to it. It also says each setting is independent of the others.
Anthropic draws the same line. It says ClaudeBot collects content that could contribute to model training. Claude-SearchBot indexes pages to improve search results, and blocking it keeps your pages out of that index. Claude-User fetches a page when someone asks Claude a question, and Anthropic says blocking it can reduce your visibility in user-directed search.
Perplexity says PerplexityBot surfaces and links websites in its search results and is not used to crawl content for AI foundation models. It recommends allowing it. Perplexity-User fetches pages for a user's question and, in Perplexity's words, generally ignores robots.txt rules. Google says Google-Extended does not impact a site's inclusion in Google Search and is not a ranking signal. It controls use of your content for Gemini training and grounding, and it has no user agent of its own.
What the pasted block really switched off
Sort the eleven lines by job and the damage is easy to see. GPTBot, ClaudeBot and Google-Extended are about training. Blocking them did what the thread wanted and nothing more. OAI-SearchBot, Claude-SearchBot and PerplexityBot are the search crawlers. Blocking those is what removed the site from three assistants' answers. The three user-request fetchers mostly did nothing, because two of the three vendors say those fetchers may not follow robots.txt anyway.
So the founder got almost none of the protection he wanted and paid for it with the one channel his buyers had started to use. His product never changed. What changed is that the assistants could no longer read the only pages that describe it, and a competitor with a thinner product was easy to read.
Launches are where this goes wrong, because nobody opens robots.txt on a deadline. When Chariot moved its speech model from private enterprise access to a public research preview, its site went from kickoff to public launch in seven weeks. On a schedule like that, the crawler file belongs on the launch checklist next to page titles and the sitemap, with someone who knows what each line does.
Why an unknown startup should let the training bots in too
Here is the part people will argue with. For most seed-stage companies, blocking GPTBot and ClaudeBot is also the wrong call. The training worry is real for publishers whose text is the product: a news site, a paid course, a research archive. Your homepage, pricing page and docs are not the product. They are marketing. You want them read and repeated.
Not every assistant answer runs a live search. When it does not, the model answers from what it learned in training. For a company it never read, that is nothing, or worse, a guess built from a directory listing. Letting the training bots read a clear page that says what you do is one of the few ways to shape that answer at all.
Block training on the paths that hold what you sell, not on the whole site. If your startup publishes a paid benchmark report or a dataset, disallow that folder for GPTBot and ClaudeBot and leave the marketing pages open. That is a business decision you can explain to an investor. A blanket block copied from a thread is not.
The switch outside robots.txt
Your host or CDN may block bots before robots.txt is ever read. Cloudflare's current documentation, updated this month, describes AI crawler policies split by behaviour: Search, Agent and Training, each with Block, Block on pages with ads, or Allow. It says new domains start on Allow for all three. An older domain, or a setting a teammate switched on when the topic was in the news, can still be set to Block, and robots.txt will not show it.
Ask whoever owns DNS to screenshot that page for you. If Search says Block, your robots.txt file can be perfect and the search crawlers still get turned away at the edge.
The fifteen-minute check to run this week
- Open yoursite.com/robots.txt in a browser. Search the page for OAI-SearchBot, Claude-SearchBot and PerplexityBot. If any of them sits above Disallow: /, delete those lines.
- Decide on training on purpose. Leave GPTBot, ClaudeBot and Google-Extended allowed for marketing pages. If you sell content, add a Disallow line for that one folder only, such as Disallow: /reports/.
- Open your CDN or host bot settings. Confirm the Search policy is set to Allow, and write down who can change it.
- View the source of your homepage and check that your one-line description is in the HTML. If the page only appears after JavaScript runs, the crawlers you just allowed may see an empty page.
- Wait a day. OpenAI says ChatGPT search can take about 24 hours to adjust to a robots.txt change. Then ask ChatGPT, Claude and Perplexity the question your buyers ask, and save each answer with the date.
- If you are still missing once the crawlers can read you, the problem has moved from access to content. That is the job covered in how to get your startup cited in ChatGPT answers.
For the vet scribe, the fix is a deleted block of lines, a one-folder rule it does not even need yet, and a homepage sentence that names veterinary clinics in its first ten words. Nothing about the product changes.
An assistant can only name you if it can read you, and then only if the page it reads says plainly what you do and who it is for. Studio Maydit designs and builds websites for AI founders in Framer, Webflow and custom code, as fixed-scope projects of three to four weeks. Whoever builds yours, make the crawler file and the first sentence of every page part of the brief, not an afterthought. If you want a second pair of eyes on your site before your next launch, book a call with Studio Maydit.
