How to Check If AI Crawlers Can Actually Reach Your Site

We found this on our own site. OpenAI’s training crawler, GPTBot, was getting HTTP 429 on every request that was not already cached. Our host later confirmed in writing that the rule is deliberate, applies across their whole platform, and cannot be turned off on a shared plan. Nothing in our configuration caused it, nothing in our logs showed it, and the site looked perfect in a browser.

Here is how to find out whether the same thing is happening to you, in about five minutes.

The test: can AI crawlers reach your site right now?

Four requests. Same machine, same minute, same page. The only thing that changes is who you say you are. That is the whole design: if a browser gets a 200 and a named AI crawler gets a 429 from the same IP in the same minute, the difference cannot be your network, your DNS or the page itself.

# 1. GPTBot, full versioned string, cache-busted
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  "https://YOURSITE.com/?miss=$RANDOM" | head -n 1

# 2. Control: an ordinary browser
curl -sI -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/139.0" \
  "https://YOURSITE.com/?miss=$RANDOM" | head -n 1

# 3. Control: OpenAI's other crawler, the one behind ChatGPT search
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" \
  "https://YOURSITE.com/?miss=$RANDOM" | head -n 1

# 4. Control: Googlebot
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  "https://YOURSITE.com/?miss=$RANDOM" | head -n 1

These are the strings we tested on 14 September. OpenAI’s bot documentation, read on 30 October 2026, now shows GPTBot/1.4 and OAI-SearchBot/1.4, and Hostinger describes its rule as matching GPTBot/1.x, so use the version OpenAI currently publishes. On 30 October, GPTBot/1.4 and GPTBot/1.2 both returned 429 on six cache-busted requests each, while an ordinary Chrome string got a 200.

If all four return 200, you are fine. If one crawler returns 429 or 403 while the browser returns 200, the refusal is keyed to that user-agent string and it is happening somewhere between you and the crawler.

Three ways this AI crawler access test goes wrong

Forgetting the random query string. Without ?miss=$RANDOM you are asking the CDN for a page it already has, and it will hand over the cached copy without ever consulting the rule that would refuse it. You will get a clean 200 and learn nothing. The cache-buster is the entire test.

Testing in a burst. This one has caught us twice, and the second time is the more useful story. On 15 September we fired 78 rapid requests as GPTBot and 34 came back refused, which looked like a partial block. On 21 September an unpaced probe made it look as though the limit had roughly halved, and we had started writing that up. It had not: the instrumented run from a clean start returned 45 of 77 readable pages, within one page of the week before. Re-requesting the refused URLs one every 20 seconds got 3 of 6 through, and one every 60 seconds got 2 of 6. A rate limit punishes exactly the behaviour a fast test performs, so a burst measures your test rather than your site. Space the requests out, and treat any figure from a parallel sweep as a floor.

Trusting one result. Answers vary. Run each identity about ten times with gaps before you believe either outcome.

What the response headers tell you

When a request is refused, read the headers rather than guessing. The status code tells you that something said no. The cache header tells you whether the request reached your origin at all, and the CDN request identifier is the one piece your host needs before they will look at a ticket. Capture all three:

curl -sI -A "<the crawler UA>" "https://YOURSITE.com/?miss=$RANDOM"

A Server value naming a CDN, a request ID and no cache status together mean the edge refused the request before your own server was involved. That distinction matters, because it tells you no amount of editing your site will fix it.

The fix you control: warm your own cache

If the refusal is at your host’s edge, you cannot remove it. What you can do is make sure your pages are already cached when a crawler arrives, because a cached response is served without consulting the rule.

The trap is that most deploy processes purge the CDN cache, which is correct for freshness and terrible for crawler access. Straight after a deploy every page is a miss, so your newest content is the least readable content you own.

The fix is one pass over your own sitemap with a normal browser user-agent, immediately after the purge:

# for each URL in your sitemap, one ordinary request, a few at a time
curl -s -o /dev/null -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/139.0" "$URL"

Measured on our own site on 15 September: after a single warming pass over the sitemap, 44 of 78 pages were readable to GPTBot, and that number is a floor rather than a ceiling because the verification itself was a burst. Re-measured the same way on 21 September: 45 of 77, with no refusal on any page the CDN served from cache. Every refusal arrived on a cache miss and every page that passed came from cache, which is the mechanism stated as plainly as a measurement can state it. Before we raised the CDN cache lifetime on 12 September, the hit ratio was 6.3%, so roughly 94% of the site was unreachable to GPTBot.

Two things worth doing alongside it. Raise your CDN cache TTL so warmed pages stay warm longer, and re-run the test weekly, since a platform rule can change without anyone telling you.

Check if GPTBot can crawl your site, then decide how worried to be

If unrestricted crawler access genuinely matters to your business, shared hosting is the wrong place to be, because the rules there are not yours. A VPS or dedicated server gives you control over server-level crawler rules. That is a real cost and a real migration, so it is worth being honest about whether it matters for you.

For most sites it does not, and here is the reassuring part. OpenAI runs three separate crawlers. GPTBot builds the training corpus. OAI-SearchBot builds the index behind ChatGPT search, and ChatGPT-User fetches a page live when someone’s question triggers browsing. On our site the last two were never refused once. So ChatGPT could still find, read and quote us in real time. What we lost was being part of what the model knows without searching, which matters over years rather than weeks.

Can AI crawlers reach a real estate agent’s website?

Check it rather than assume it. Many small agent sites run on shared hosting, or on a website platform supplied by a brokerage or an IDX vendor, where the edge rules belong to someone else. Run the four requests against your homepage, one agent bio, one neighborhood page and your newest listing, because a listing published this morning is a brand-new URL and every new URL starts as a cache miss.

Three things are specific to agent and brokerage sites:

• Listings may not be in the HTML at all. Some IDX setups load the listing body with JavaScript after the page arrives. Fetch a listing with curl, and if the description and price are missing from the raw response, a crawler is reading an empty frame.
• You may not control the rule. On a hosted agent platform, as on our own shared host, the setting that refuses a crawler can sit with the provider. Ask them in writing which AI crawlers they rate-limit or block, and keep the answer.
• Your own site is one source among several. In our 500-query Houston study, a firm’s own domain was cited in 8.4% to 10.4% of the AI answers naming it, while portals and agent referral networks took 43.6% of all citations.

So treat crawler access as the floor of the job. Fix it first, because nothing on your site can be cited without it, then put the same care into the portal profiles and listings the engines read alongside you. Our sample AI visibility audit for real estate shows where the server-side crawler check sits among the other checks, and the llms.txt guide covers the plain-text map of your key pages that AI systems can read. Our guide to how to rank in ChatGPT covers which OpenAI crawler controls search and which controls training.

---

The full write-up of our own case, including the measured results, the response headers with CDN request IDs, our host’s confirmation in their own words, and the raw JSON from every run, is published here: Hostinger’s CDN blocks GPTBot. All of it is CC BY, so you are free to reuse the numbers with a link.

If you would rather have someone else run this for you, our Free AI Visibility Audit checks crawler access alongside whether ChatGPT, Perplexity, Gemini and Copilot actually name you, and emails the report inside 48 hours.