Which robots.txt group applies to each crawler?
The Robots Exclusion Protocol, published as RFC 9309, sets the rules compliant crawlers follow. The file must sit at /robots.txt in the top-level path of each host. A crawler matches its product token to the user-agent lines without regard to case and obeys the matching group. If two groups match the same token, their rules are combined. Only when no group matches does the crawler fall back to the group for *. Within a group, the most specific rule, meaning the longest matching path, wins. The RFC also says these rules do not authorise or deny access; they are requests that compliant crawlers honour.
Hypothetical example: a site has a * group that disallows /admin/ and a separate GPTBot group that disallows everything. GPTBot follows its own group and stays out. OAI-SearchBot has no group of its own, so it follows the * group and may crawl everything except /admin/.
- Search crawlers: Googlebot, OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- Training crawlers: GPTBot and ClaudeBot.
- User-initiated fetchers: ChatGPT-User, Claude-User and Perplexity-User.
Does robots.txt itself load correctly?
Open https://yourdomain/robots.txt in a browser and check its status code with a header checker or curl -I. RFC 9309 treats status codes differently. A 4xx response such as 404 means the file is unavailable, and crawlers may then access any page. A 5xx server error means the file is unreachable, and crawlers must assume everything is disallowed. A site whose robots.txt sometimes returns 500 or 503 can lose crawler access without anyone editing the file. Crawlers may cache the file, and the RFC says a cached copy should not be used for more than 24 hours unless the file is unreachable.
Each hostname needs its own file. Anthropic's documentation, for example, tells site owners to add robots.txt rules for every subdomain they want covered, so check www, non-www and any booking or shop subdomain.
Is the CDN or firewall blocking before robots.txt?
A crawler that receives a 403 or a challenge page never gets to read your robots.txt, so a firewall setting can override a file that allows everything. Cloudflare now groups AI traffic into three behaviours that each site controls separately: Search, for crawlers that index content to answer questions later; Agent, for automated activity on a person's behalf; and Training, for crawlers that train or fine-tune models. Each can be set to block on all pages, block only on pages with ads, or allow. The settings are under Security Settings, then Configure AI bot policies.
Cloudflare says new domains onboarded from 15 September 2026 block Training and Agent bots on pages that display ads while Search stays allowed, and that crawlers used for both search and training are caught by training blocks. Check this setting on any recently added domain, and check rules on other hosts that block countries or data centres.
How do you request a page as each crawler?
From a terminal, request a key page with a crawler's user-agent token and compare the status code and page size with a normal browser request. For example: curl -s -o /dev/null -w '%{http_code}' -A 'PerplexityBot/1.0' https://www.example.com.au/services/. A 200 that matches the browser result is a pass. A 403, a 429 or a much smaller response suggests a rule is catching that agent. The test has a limit: some bot management products check the source IP address as well as the user agent, so a request from your own computer may be treated differently from the real crawler. Use it to find obvious blocks, then confirm in the logs.
What do the server logs show?
Server or CDN logs show what the real crawlers received. Search the access logs for each token, such as OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot, and look at the status codes returned. Then check the requesting IP addresses against the lists the operators publish, because anyone can copy a user-agent string. OpenAI and Perplexity publish JSON files of their crawler addresses, and Anthropic's crawler page links to its list. A run of 403 responses to a verified crawler address is the clearest sign of an unintended block.
- OpenAI: openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json.
- Perplexity: perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.
- Anthropic: the IP address list linked from its crawler help page.
What about agents that ignore robots.txt?
Some agents fetch a page because a person asked a question, and operators treat robots.txt differently for them. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says its bots respect robots.txt directives, and Claude-User can be controlled there. Stopping a fetcher that ignores robots.txt has to be done at the firewall with the published IP lists. Anthropic warns that blocking its IP addresses may not work as a lasting opt-out because it stops the crawler reading your robots.txt. For a business that wants to be recommended, allowing these fetchers is the simpler choice.