Documentation
PaceghostBot
What is PaceghostBot?
PaceghostBot is the official crawler used by Paceghost.io, a platform that helps site owners monitor and improve how their websites are seen by AI systems, search agents, and large language models.
We use this bot to analyze public pages on request and simulate how AI crawlers interpret your content.
User Agent
PaceghostBot identifies itself using the following user agent string:
PaceghostBot/1.0 (+https://www.paceghost.io/resources/paceghostbot/)You can use this string in your server logs or analytics tools to identify and monitor our visits.
How Audits Fetch Your Page
The crawlability check uses the PaceghostBot/1.0 crawler above. For the snapshot step of an audit, we fetch your page through a ladder of fetch approaches, trying each in turn:
Disclosed bot — available to every audit
A headless browser that identifies itself openly with the following user agent:
Mozilla/5.0 (compatible; Paceghost/1.0; +https://www.paceghost.io/resources/paceghostbot/)Genuine browser — verified owners
A real, headed Chromium with no user-agent override — it appears to your site exactly as a regular visitor's browser would. This approach is available when you have verified ownership of the domain.
The approach that succeeded is remembered per domain, so later audits start with the approach that worked. Which fetch client served an audit is disclosed in the report.
If your site's security layer refuses every allowed approach (HTTP 401, 403, or 429), the audit reports that block as a finding rather than auditing the error page. To let audits through, allow the disclosed user agent above in your CDN or firewall bot rules.
Privacy and Scope
Audits scan only websites that users request on our platform. The one exception is the research study below, which reads three public files. From these files we keep only the parsed rules, not the files. We also keep the host names of the websites we read, and a host name can contain the name of a person. Our privacy policy explains what we keep and for how long.
- Multi-page snapshots fetch pages from a list derived in advance from your sitemap, llms.txt, and homepage — never by following links we find on your pages
- We do not crawl pages blocked by robots.txt
When you run a multi-page snapshot, we fetch and store the text of the pages we sample as part of your report — so the analysis is reproducible and auditable. Page text is deleted together with the scan after 24 months.
Research study on AI-crawler rules
For a study on how websites treat AI crawlers, PaceghostBot reads three public files from popular websites: /robots.txt, /llms.txt and /ai.txt. It fetches no other pages. The list of websites comes from the public Chrome UX Report: the most visited websites in Germany, Austria, Switzerland and worldwide.
For the study, the user agent contains the word study. It is not PaceghostOutreachBot, our separate outreach crawler.
PaceghostBot/1.0 (study; +https://www.paceghost.io/resources/paceghostbot/)- We read robots.txt first. If it disallows PaceghostBot, we fetch no other file from your site.
- If your robots.txt has its own group for PaceghostBot that disallows it, we leave your site out of the results. We only count how many sites we left out.
- We send at most one request per second to a site.
- We keep only the rules we parse from the files, not the files themselves.
- We publish aggregate figures only, for example the share of sites that block a given AI crawler. We do not name single websites.
To leave the study, write to info@paceghost.io with the host names of your websites. We then fetch no file from them for the study. Audits of your websites stay possible.
You can also add a User-agent: PaceghostBot group with Disallow: / to your robots.txt as shown below. This group blocks PaceghostBot completely: it also stops PaceghostBot from reading your pages when a user requests an audit of your site.
How to Block PaceghostBot
If you prefer not to allow PaceghostBot to access your site, simply disallow it via your robots.txt file.
Example:
User-agent: PaceghostBot
Disallow: /Our bot respects this directive. The directive applies to audits and to the research study. To leave only the study, write to info@paceghost.io.
Contact
If you have questions or concerns about PaceghostBot, reach out to: info@paceghost.io