Access for AI services

What we check

We check in three ways whether AI services can read the website:

1. robots.txt. We read whether it blocks crawlers that deliver AI answers:

Service	Crawler
ChatGPT	OAI-SearchBot, ChatGPT-User
Claude	Claude-SearchBot, Claude-User
Perplexity	PerplexityBot, Perplexity-User
Copilot	Bingbot
Gemini	Googlebot

If one of them is blocked, that is a serious defect. Smaller assistants reach fewer people, so blocking them costs only half of the points:

Service	Crawler
Meta AI	Meta-ExternalFetcher
Siri and Apple Intelligence	Applebot
DuckDuckGo	DuckAssistBot
Alexa	Amazonbot
Le Chat (Mistral)	MistralAI-User

Blocked training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider) we report only as a note without deducting points.

2. Firewall. We request the home page with the identifiers of AI services. If the server answers with an error or a bot check although it otherwise shows us the page normally, bot protection is probably blocking the AI services. Because some firewalls also check where a request comes from, this is a pointer, not proof, and costs only half of the points.

3. Content Signals. We read whether the robots.txt uses "Content-Signal" to define what AI services may use the content for. If it prohibits search or AI answers (search=no or ai-input=no), there is a warning.

Why it matters

More and more people ask ChatGPT, Claude, Perplexity or Copilot instead of Google: "Which bank offers an account without account fees?" The assistants then search the web and quote the pages they were able to read.

A website that shuts these services out does not appear in the answers. This often happens unintentionally:

• An agency or a plugin set "block all AI bots" to protect content from training, and caught the answer crawlers as well.
• Bot protection such as Cloudflare blocks AI crawlers by default.
• A robots.txt template from the internet blocks everything except Google across the board.

We measure whether AI services can read the website. Whether they actually recommend the website or the organisation behind it depends on many further factors.

How to fix it

robots.txt

A robots.txt that allows answer crawlers and blocks training crawlers looks like this:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow:

Sitemap: https://www.example.ch/sitemap.xml

If you also want to allow training, leave out the first block and set ai-train=yes.

Bot protection and firewall

Check in Cloudflare, Sucuri, Wordfence or the firewall of the hosting provider whether AI crawlers are blocked. Allow at least the answer crawlers from the table above.

WordPress

SEO plugins such as Yoast SEO and Rank Math have an editor for the robots.txt. Also check security plugins for settings such as "Block AI bots".

How to check it yourself

Open https://example.ch/robots.txt and look for the names from the table. If Disallow: / is under one of them, it is blocked.

Frequently asked questions

I do not want AI companies to train on my texts. Do I have to block everything?

No. For training there are crawlers of their own such as GPTBot, ClaudeBot or Google-Extended. If you block these, it costs no points. The crawlers that deliver answers and recommendations are separate from them and can keep access.

What is the difference between a training crawler and an answer crawler?

Training crawlers collect texts to train future AI models. Answer crawlers open pages when someone asks a question right now and quote them with a link. Only the latter bring visitors directly.

I use Cloudflare. What do I have to set?

Cloudflare can block AI crawlers through "AI Crawl Control" or the bot settings. Check there which crawlers are blocked and allow at least the answer crawlers.