How websites in Germany, Austria and Switzerland treat AI crawlers

Key figures

• GPTBot is the AI crawler that websites block by name most often: 10.7% in DACH and 14.4% in the global group. Claude-User is blocked least: 1.9% and 3.4%.
• Switzerland has the highest share of websites with AI rules (28.4%), then Austria (24.0%) and Germany (22.3%).
• In Germany, the top 1,000 websites have AI rules 2.2 times as often as the websites ranked 5,001 to 10,000 (40.4% against 18.1%). In the global group the ratio is 1.3.
• Few websites contradict their own robots.txt: 6 of 1,601 websites that have both a robots.txt and an llms.txt or ai.txt.

Which AI crawlers are blocked by name

The share of websites with a robots.txt that have a User-agent group for the crawler, and that group disallows the whole site (Disallow: /). The table is sorted by the DACH column.

In DACH and the global group, between 1.3% and 4.8% of the websites name a crawler and allow it. Most websites do not name the crawler at all. For that crawler, the User-agent: * group of the file applies (RFC 9309 ↗). The CSV also has the share that blocks each crawler through the * group.

How many websites have AI-specific rules

An AI-specific rule is a User-agent group for at least one of the 15 crawlers, with Allow or with Disallow. A website counts as having a robots.txt only when the file is a real robots.txt.

By rank

The Chrome UX Report sorts websites into rank buckets, not exact ranks. The share of websites with a robots.txt that have any AI rule:

The lists for Austria and Switzerland end at 5,000. The DACH bucket 5,001 to 10,000 therefore holds German websites only.

llms.txt and ai.txt

The share of websites that serve a real llms.txt or ai.txt. A file counts only when the server answers with status 200 at the exact path, and the body has the format of the file. An llms.txt starts with a Markdown heading (# Name). An ai.txt has robots.txt rules or Field: value lines.

This check matters. Some servers answer a missing file with their HTML home page and status 200. A check that only asks for a successful answer with a body would count 2,540 DACH websites with an llms.txt, not 1,262. That is 2.0 times too many. Of the 28 websites with a real ai.txt, 4 use Field: value lines and no robots.txt rules.

Contradictions with robots.txt

This section counts websites where llms.txt or ai.txt says something different from robots.txt. Of 1,601 websites with a real robots.txt and a real llms.txt or ai.txt, 6 (0.37%) contradict their robots.txt. In 5 of them, robots.txt blocks all 15 AI crawlers while the website publishes an llms.txt. In 1 of them, the ai.txt allows a crawler that robots.txt blocks. The study cannot check whether an llms.txt links to pages that robots.txt blocks, because it keeps no links.

An earlier study

In May 2026, Thunderbit ↗ read the robots.txt files of the Tranco top 10,000 and reported that 17.0% of the sites have an explicit AI block. Thunderbit counts every robots.txt with status 200 plus every 404. On the same base, the global group of this study gives 16.1% (base: 7,609 websites). The two numbers are not a trend: the website lists, the dates and the crawler lists differ. Thunderbit's bot list names 22 AI crawlers, this study uses 15.

Last checked: October 13, 2026.

Method

1. Sample: the CrUX rankings of August 2026 for Germany and the world (to rank bucket 10,000) and for Austria and Switzerland (to rank bucket 5,000). CrUX lists the websites that Chrome users visit, by origin. Google publishes the rankings in BigQuery ↗, in the tables chrome-ux-report.experimental.country and chrome-ux-report.experimental.global. We read the four lists from these tables on October 14, 2026, with the filters yyyymm = 202608 and experimental.popularity.rank <= 10000 (5,000 for Austria and Switzerland). A table can list one origin more than once in a month, so we kept each pair of origin and rank once. To query BigQuery, you need a Google Cloud project. After duplicate hosts are removed, the sample has 21,761 hosts: 15,051 in a DACH list and 9,998 in the global list. 3,288 hosts are in both, so the two groups overlap. The host list for the fetch on October 13, 2026 came from a copy of the same lists. Our BigQuery read on October 14, 2026 matched that copy exactly.
2. Fetch: on October 13, 2026, PaceghostBot fetched /robots.txt first, then /llms.txt and /ai.txt when robots.txt allowed it. It sent at most one request per second to a website and fetched no other pages. Its rules and its user agent are on the PaceghostBot page. No website asked to leave the study, and no robots.txt blocked PaceghostBot by name.
3. Parser: robots.txt is read with the same code that our audit uses. A crawler counts as blocked when its own group disallows the root path /. A rule for one folder does not count as a block.
4. Base: a website has a robots.txt only when the server answers /robots.txt with status 200 and a file that has robots.txt directives. An HTML page, an empty answer or a redirect to another path does not count. DACH: 11,751 websites. Global: 6,320. For llms.txt and ai.txt, the base is the websites where PaceghostBot was allowed to request the file and got an answer.
5. In the tables, "–" means that the list has no such rank bucket.
6. Limits: the data shows the state of one day. In the global group, more servers refused the request for robots.txt or did not answer (23.1% against 10.8% in DACH), so the study read fewer robots.txt files there. The 15 crawlers are not every AI crawler.

The crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot.

Data and license

The CSV has one row for each group, rank bucket, measure and crawler, with the count, the base and the share in percent. It holds aggregates only and names no website. The *_success_with_body rows count every successful answer with a body, before the content check. The license, the sources and a description of each column are in a metadata file in the CSV on the Web (CSVW) format.

License: CC BY 4.0 ↗. Please cite "paceghost.io, AI crawler rules study, October 2026" with a link to this page. Sources: the Chrome UX Report ↗ by Google, licensed under CC BY 4.0. We changed the data: we combined the four lists, removed duplicate hosts, read three files from each host and counted the results. The CSV holds these counts only, not the website lists.