Skip to content
Traiker
ES

Should your company let AI bots into its website?

Traiker Team

26 October 2026 · Updated on 29 October 2026 · 3 min read

Illustration: an open garden gate, a paper plane flying through it, a signpost and a lit lantern

In short

GPTBot, ClaudeBot, OAI-SearchBot… Not all AI bots do the same thing. We explain which ones you should let through, how you might be blocking them without knowing it and why at Traiker we have opened our website to them.

In this article
  1. In short
  2. Which AI bots visit your website?
  3. Could you be blocking them without knowing it?
  4. Why do we let training bots in at Traiker?
  5. Doesn't that clash with “we don't train on your data”?
  6. Frequently asked questions

In short

It depends on which bot. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) should almost always be let through: if you block them, you disappear from the answers. Training bots (GPTBot, ClaudeBot, Google-Extended) are a business decision. At Traiker we have opened the door to all of them on our public website, and we explain why.

Which AI bots visit your website?

Each AI company uses several bots, and they are controlled separately in robots.txt:

  1. Training bots. GPTBot (OpenAI), ClaudeBot (Anthropic) and Google-Extended (Google) collect content that may be used to train models.
  2. Search bots. OAI-SearchBot, Claude-SearchBot or PerplexityBot index pages so they can be cited when someone asks.
  3. Bots acting on a user's behalf. ChatGPT-User or Claude-User open a page when a person asks them to in the chat.

According to OpenAI's documentation, each setting is independent: you can allow OAI-SearchBot to appear in ChatGPT searches and block GPTBot so your content isn't used for training. Google says the same about Google-Extended: it doesn't affect your presence in Google Search or act as a ranking signal. And Anthropic warns that blocking Claude-SearchBot can reduce your visibility in Claude.

Could you be blocking them without knowing it?

Yes, and it happened to us. In traiker.com's robots.txt we found a block we hadn't written: Cloudflare, our network provider, was adding it, and it denied entry to GPTBot, ClaudeBot, Google-Extended and other training bots. On top of that, those bots received a 403 error before reaching the website.

Cloudflare's documentation explains it: if the training policy is set to “Disallow” with preference syncing on, Cloudflare puts its rules ahead of your robots.txt and blocks those bots at the edge. Your file can say one thing and your website do another: open yourdomain.com/robots.txt from time to time and read it all the way through.

Why do we let training bots in at Traiker?

Because we are a new brand and we want the models to know we exist. A University of Toronto study of 158,099 AI search-engine answers found that ChatGPT includes a brand in its searches in 67% of cases when it already knows it, compared with 11% when it doesn't. And 46% of the time it names a brand the user hadn't mentioned, there is no cited page backing it up: it pulls it from memory.

The same study warns that these are associations, not cause and effect, and that in a market of small B2B vendors the model barely knew the brands. Exactly our case. So on 23 October we removed the Cloudflare block and left explicit permission in our robots.txt for search, user and training bots.

Doesn't that clash with “we don't train on your data”?

No, because they are different things. Our public website is made to be read by anyone, person or machine. The data you store in Traiker is not public: each person or company has their own encrypted database. Your data is yours: we don't train models on it or process it. Your assistants only query it when you connect them through MCP; we're opening access gradually in the beta.

If what AI does with your conversations worries you, you'll find it in what ChatGPT, Claude and Grok do with what you tell them. And if you're thinking about how to bring your company into AI, start with what it means to be an AI First company.

Frequently asked questions

Does blocking GPTBot remove me from ChatGPT?

Not from its searches, according to OpenAI, if you let OAI-SearchBot through. You are only saying you don't want your content used for training.

Do bots always respect robots.txt?

There is no guarantee. Cloudflare reminds you that complying is voluntary, and OpenAI says that on visits started by a user (ChatGPT-User) it may not apply.

How long does a change take to show up?

OpenAI estimates around 24 hours for its search to adjust to a new robots.txt. For training, it only affects what they collect from that moment on.

If you want your important information to be in a place of your own and your assistants to work with it, sign up for the Traiker beta.

Keep reading