Should You Block or Allow AI Crawlers on Your Store? (The GPTBot Myth)
Blocking GPTBot stops model training, not ChatGPT Shopping. See what each AI crawler controls and what your eCommerce store should actually block in robots.txt.
You found a robots.txt snippet titled "block every AI bot," pasted it into your store, and moved on. Now you're not sure whether you just hid your catalog from ChatGPT. Here's what each AI crawler actually controls, whether blocking one costs you product visibility, and the policy most stores should run.
Blocking GPTBot stops OpenAI from using your store's pages to train its models. It does not remove your products from ChatGPT Shopping or ChatGPT search. Those run on your product feed, your structured data, and a separate crawler called OAI-SearchBot. Block the training bot if you have a reason to. Keep the answer-engine bots open.
TL;DR: Blocking GPTBot only opts your store out of OpenAI model training. It doesn't remove your products from ChatGPT Shopping, which runs on merchant product feeds and Product structured data, not GPTBot's crawl. The crawler that controls ChatGPT search visibility is OAI-SearchBot. For most stores the right policy: decide on training bots on purpose, keep answer-engine bots open, and block cart, checkout, account, and filter URLs regardless.
Key Takeaways
- GPTBot, OAI-SearchBot, and ChatGPT-User are three of OpenAI's documented crawlers, each with its own robots.txt control — training, search surfacing, and user-triggered fetch (OpenAI, 2026).
- Opting out of OAI-SearchBot removes a site from ChatGPT search answers. Opting out of GPTBot does not (OpenAI, 2026).
- ChatGPT Shopping results are built from merchant product feeds and Product or Offer structured data, plus OAI-SearchBot crawling — not from GPTBot's training crawl (OpenAI Help Center, 2026).
- Blocking Google-Extended has no effect on Google Search rankings or AI Overviews. Google states it is not a ranking signal, and AI Overviews use the live Googlebot index (Google Search Central, 2026).
- robots.txt is a request, not a wall. A CDN or WAF rule is what actually enforces a block — and an over-broad one is a common way stores lock out answer engines by accident.
What does blocking GPTBot actually do?
Blocking GPTBot removes your pages from OpenAI's model-training data. That is the entire effect. It doesn't touch ChatGPT's live search or its shopping features.
OpenAI documents four crawlers. Three decide what happens to your store's ordinary pages, and each has its own line in robots.txt. GPTBot collects pages that may be used to train OpenAI's foundation models. OAI-SearchBot surfaces sites inside ChatGPT's search features. ChatGPT-User fetches a page when a person — or an agent acting for them — asks ChatGPT to open it. The fourth, OAI-AdsBot, only checks pages a merchant submits as ads, so it sits outside this decision. You can allow one and block another.
Because the controls are independent, a disallow rule for GPTBot says nothing about the other two. OpenAI notes that if you allow both GPTBot and OAI-SearchBot, it may use the results from just one crawl for both to avoid fetching your pages twice (OpenAI, 2026). One more detail matters for stores: OpenAI states that robots.txt rules may not apply to ChatGPT-User, because that fetch is triggered by a person rather than by an automated crawl.
Three of OpenAI's crawlers carry separate robots.txt controls: GPTBot for model training, OAI-SearchBot for search surfacing in ChatGPT, and ChatGPT-User for user-triggered fetches. Blocking one has no effect on the others (OpenAI, 2026).
Does blocking GPTBot remove my products from ChatGPT Shopping?
No. ChatGPT Shopping results come from merchant product feeds and Product structured data, plus OAI-SearchBot crawling. None of that depends on GPTBot.
ChatGPT builds a product result from a few inputs. Merchants can submit a structured product feed straight to OpenAI — accepted formats include CSV, TSV, XML, and JSON — and keep it refreshed so price and availability stay current (OpenAI Help Center, 2026). ChatGPT also reads Product, Offer, and review markup on the product pages it crawls, pulls from third-party product-data providers, and, for Shopify stores, uses catalog data already integrated with OpenAI.
OpenAI says product results are organic and unsponsored, chosen for relevance rather than sold as ads, and that when several merchants sell the same item they are ranked on signals like availability, price, quality, and whether you are the maker or primary seller (OpenAI Help Center, 2026). Every lever on that list is a data lever: feed completeness, attribute accuracy, structured data, stock and price freshness. This is what Answer Engine Optimization (AEO) for commerce actually means — getting your product data recommended, not your brand mentioned. Crawl permission for GPTBot is not on the list.
The training crawl and the shopping pipeline are separate systems. A store that blocks GPTBot but keeps a complete feed and full schema stays eligible for the same carousels as one that allows it. If your products are missing from ChatGPT Shopping — or ChatGPT keeps recommending a competitor instead of you — the cause is almost always feed or schema, not a crawler line. See how ChatGPT Shopping actually selects products, then audit your feed for completeness and schema accuracy.
What about Google — does Google-Extended affect my rankings or AI Overviews?
No. Google-Extended controls whether your content trains Gemini and grounds some generative features. Google states it is not a ranking signal and does not affect your inclusion in Search or AI Overviews.
Google-Extended is a robots.txt token, not a separate crawler. Google states it "doesn't have a separate HTTP request user agent string" — Googlebot still does the crawling. Blocking Google-Extended tells Google not to use what it crawled to train future Gemini models or to ground Gemini Apps and Vertex AI, and nothing else. Google is explicit that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google Search Central, 2026).
AI Overviews and AI Mode are Search features. They draw on the same live index as the blue-link results, and a page has to be indexed and snippet-eligible to appear in them at all. The controls that limit how your content shows up there are the snippet directives: nosnippet, data-nosnippet, max-snippet, and noindex. Google-Extended is not one of them.
The real footgun is blocking Googlebot itself. Do that and you drop out of Search and AI Overviews at once. For why strong Google rankings still don't guarantee AI visibility, see why Google's ranking signals don't carry into AI answers.
So should an eCommerce store block AI crawlers?
For most stores: keep answer-engine crawlers open — OAI-SearchBot, PerplexityBot, Googlebot — decide on training crawlers deliberately, and monitor agent traffic rather than block it.
| Bot class | Examples | Default for most stores | Block only if… |
|---|---|---|---|
| Answer engine / search | OAI-SearchBot, PerplexityBot, Googlebot | Allow | You don't want AI-referred shoppers (rare) |
| Model training | GPTBot, ClaudeBot, CCBot, Google-Extended | Your call | You have licensing or IP concerns about training use |
| User / agent traffic | ChatGPT-User, Perplexity-User | Allow, monitor | Logs show abuse or cost spikes |
The tradeoff isn't symmetric. Blocking a training bot has a small, slow, hard-to-measure downside: less of your catalog in the model's background knowledge over time. Blocking an answer-engine bot has an immediate, measurable one. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers," though they can still appear as plain navigational links (OpenAI, 2026). For a store, that means losing the product surfaces and keeping only a bare URL.
Opting out of OAI-SearchBot removes a site from ChatGPT search answers. Opting out of GPTBot does not (OpenAI, 2026).
What should every store block, regardless of AI policy?
Block the pages that were never meant to be indexed: cart, checkout, account, order confirmation, internal search results, and faceted-filter URL parameters. This is crawl hygiene, not an AI decision.
These rules apply to Googlebot and every AI crawler equally. Stores need them more than most sites because faceted navigation — color, size, price, and brand filters carried in the URL — generates near-infinite low-value pages. Every crawler that spends its budget on filter combinations is one that isn't reading your real product and collection pages.
Keep those product, collection, and content pages open to everything in the allow column above. The goal is to point every crawler, AI or not, at the pages you want recommended. For the markup those pages need, focus on complete Product, Offer, and review schema.
What blocking can't do
robots.txt is a published request, not access control. Compliant crawlers honor it; others ignore it. Real enforcement happens at the network layer.
Major AI companies, including OpenAI and Google, state that they respect robots.txt. Enforcement is still voluntary, and non-compliant scrapers exist. If you need a hard block, that is a job for a CDN or WAF rule — Cloudflare bot controls, for example — which rejects the request before it reaches your store.
That same power cuts the other way. An over-broad CDN bot rule, or a stray blanket disallow left in a staging robots.txt that got promoted to production, is a common way stores block OAI-SearchBot, PerplexityBot, or Googlebot without meaning to. The deliberate GPTBot block is rarely the problem. The copied snippet nobody audited usually is.
Practical check: pull your live robots.txt, confirm every user-agent block is intentional, and test a product URL against each answer-engine bot's user-agent string. If you're not sure your store is reachable at all, work through why your products might not be showing up in AI search.
Major AI operators, including OpenAI and Google, publicly state that they honor robots.txt — but compliance is voluntary. Only a network-layer rule in a CDN or WAF enforces a block at the request itself (OpenAI and Google crawler policies, 2026).
Frequently asked questions
If I block GPTBot, will my products still show up in ChatGPT Shopping?
Yes. ChatGPT Shopping is populated by your product feed and Product or Offer structured data, plus OAI-SearchBot crawling. GPTBot is only for model training, so a GPTBot block leaves your shopping eligibility untouched (OpenAI Help Center, 2026).
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects pages that may train OpenAI's models. OAI-SearchBot surfaces your site in ChatGPT's search features. They are different crawlers with different robots.txt lines, and blocking one does not affect the other (OpenAI, 2026).
Is blocking AI crawlers bad for SEO?
Blocking training-only crawlers like GPTBot and Google-Extended has no measured effect on Google rankings. Blocking Googlebot or OAI-SearchBot does hurt visibility — in Google Search and in ChatGPT answers respectively (Google Search Central; OpenAI, 2026).
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews use the live Google index, and Google-Extended only governs Gemini training and grounding. Google states it is not a ranking signal and does not change your inclusion in Search or AI Overviews (Google Search Central, 2026).
What AI crawler settings should a Shopify store use?
Allow OAI-SearchBot, PerplexityBot, and Googlebot. Decide on GPTBot, ClaudeBot, and CCBot based on how you view training use. Block cart, checkout, account, and filter URLs for every crawler. Shopify already integrates your catalog with OpenAI, so your feed work still matters even if you block GPTBot.
Write the Policy, Don't Paste the Snippet
Three numbers to keep. Three OpenAI crawlers decide what happens to your store's pages — GPTBot, OAI-SearchBot, and ChatGPT-User. Exactly one of them, OAI-SearchBot, controls whether your store shows up in ChatGPT. And there is zero measured ranking impact from blocking training-only bots like GPTBot or Google-Extended.
The question was never "block or allow." It is "which class of bot, and on purpose." Keep answer-engine crawlers open, make a deliberate call on training crawlers, and block the cart-and-checkout clutter no crawler should see.
Then check the part robots.txt can't tell you: whether your product data is actually eligible for AI shopping surfaces. Run a free AEOsome citation and feed check to see where your store stands, independent of your crawler policy.
About the author. Vijaya Kumar Channalli is the founder of AEOsome (Tartu, Estonia), which helps eCommerce brands get their products recommended when shoppers ask AI what to buy. He writes about AI shopping visibility, product-feed compliance, and how Answer Engine Optimization differs from SEO. More at vijay.cc.
Sources: OpenAI crawler documentation, developers.openai.com/api/docs/bots; OpenAI Help Center, "Shopping with ChatGPT search"; Google Search Central, Google crawlers and Google-Extended documentation.
