Hostinger's CDN blocks GPTBot with HTTP 429 on every cache miss: the evidence, the diagnosis and the ticket

Any request that reaches Hostinger's edge with the versioned GPTBot user agent and no cached copy to serve gets HTTP 429 with an empty body. The response carries Server: hcdn and a CDN request ID and never touches the origin server, so nothing in the site's own configuration is involved. Requests that hit a warm cache pass, which is why the block hides from casual checks.

One detail worth stating exactly. The bare word GPTBot with no version and no Mozilla prefix passed 8 of 8 on this run. On 11 September the string GPTBot/1.2 on its own failed 5 of 5. The rule matches something in the real crawler's string, and OpenAI's real crawler sends the full string, so the practical effect is total.

OpenAI runs three crawlers with three jobs, and this rule blocks only one. GPTBot builds the training corpus, the part of the model that knows things without searching. OAI-SearchBot builds the index behind ChatGPT search. ChatGPT-User fetches a page live when a question triggers browsing. The last two returned 200 on every request, so ChatGPT can still find, read and quote this site in real time.

The production log settles it independently of any test we ran. In 14 to 21 September 2026, after removing our own testing IPs, OAI-SearchBot fetched 91 pages and ChatGPT-User fetched 130, while GPTBot fetched 0, because every GPTBot line in the raw file came from our own reproduction runs.

Which AI crawlers fetched nothing at all?

Three, over 14 to 21 September 2026: GPTBot, PerplexityBot, Google-Extended. A crawler that fetches nothing cannot hold current information about a site, and only one of those three is being refused. PerplexityBot and Google-Extended were free to crawl and simply did not. Over the same window Anthropic's ClaudeBot fetched 34 pages and Claude-User fetched 4 more on live questions, which is the difference between a crawler that is blocked and one that is uninterested.

A host will assume the customer's configuration is at fault, reasonably, so the job was to remove every customer-side variable by direct test before asking anyone to look at the edge. Six checks did it, and the sixth is the one that ended the argument.

1. The 429 never reaches origin. It carries Server: hcdn, a CDN request ID and Content-Length: 0. No PHP, no .htaccess, no logging script can be in the path.
2. CDN security level Medium and Off produce identical 429s. The setting is irrelevant to the rule.
3. hPanel exposes no user-agent rule of any kind. Zero IP blocks, zero country blocks. A rule that matches a user agent cannot be something the customer configured.
4. No Cloudflare in front. Nameservers and IPs are Hostinger's, and no cf- header appears anywhere.
5. Hostinger's own edge diagnostic clears the client. /hcdn-cgi/myip returns every classification flag at zero, and it returns 200 under the GPTBot identity while content paths return 429 in the same minute. The TLS fingerprint is identical across identities. The rule is on the user agent and the path, not the client.
6. A second, unrelated domain on the same host fails identically. Different site, different configuration, same 429, 8 of 8. Two unrelated domains failing the same way is not customer configuration, and Hostinger listed both in the escalation.

The response headers, request by request

Four different edge nodes (edge10, edge6, edge7, edge8, edge9) returned the same result, so it is not one misbehaving node. Every 200 is marked DYNAMIC, meaning it went to origin; every 429 has no cache status at all, because it was refused before the cache was consulted.

The diagnosis took a month and was wrong twice, and both wrong turns are in the record because they are the useful part. The first blamed our own logging detour for AI crawlers. The second was a kill switch that removed it and changed nothing. What finally worked was refusing to theorise and measuring the same request under different identities in the same second.

2026-08-12

Full crawler sweep: 16 AI user agents, all HTTP 200. GPTBot passed on a CDN cache HIT, so this test never actually reached the rule.

2026-08-22

Audit finds GPTBot receiving HTTP 429 on every cache-miss page. First attribution, to our own .htaccess routing AI crawlers through a logging script, was wrong.

2026-08-28

Kill switch removes the logging detour entirely. GPTBot still 429s identically. Origin was never the mechanism.

2026-09-01

UA-scoped, not IP-scoped: Chrome 200 and OAI-SearchBot 200 from the same IP in the same second as GPTBot 429. Response carries Server: hcdn and a CDN request ID, never reaching origin.

2026-09-11

15 consecutive cache-busted GPTBot requests, 15 of 15 returned 429. Controls in the same minute from the same IP: Chrome 15 of 15, OAI-SearchBot 15 of 15, Googlebot 15 of 15, all 200. Reproduced on a second, unrelated domain on the same host. Hostinger live chat confirms GPTBot is absent from their verified-crawler allowlist.

2026-09-12

Workaround shipped: CDN TTL raised from 10 minutes to 4 hours and Vary: User-Agent removed, so GPTBot can read pages another visitor has already warmed. Cache misses still 429.

2026-09-14

Ticket #21396332 escalated to Hostinger infrastructure. Written confirmation: platform-level edge rule matching GPTBot, no customer-side setting, no timeline. Reproduction on this page run the same day.

2026-09-15

Hostinger closes the ticket. Final answer: the 429s are intentional and server-wide, a rate limit applied specifically to OpenAI training crawler GPTBot/1.x at the infrastructure level, to protect shared server resources, not configurable per domain. No exception on shared hosting. This corrected our own earlier claim that it was not rate limiting.

Ticket #21396332, opened 2026-09-11 and escalated to the infrastructure team on 2026-09-14. The support agent's confirmation, quoted from the ticket, is the part that matters: the refusal is not a misconfiguration on this account, and no site owner can switch it off. It is a deliberate rule applied across the platform, stated by the host in its own words.

"The 429 is being generated at the Hostinger CDN edge, before your origin is ever reached."

"It affects two unrelated domains on our platform with no shared configuration."

"Security level and traffic blocking rules have no effect."

"Every other major crawler passes without issue; only GPTBot is affected."

"This points to a platform-level edge rule that is specifically matching GPTBot, something that exists at the infrastructure layer and is not configurable from hPanel. There is no setting available to you (or to us at the support level) that can whitelist a bot UA or IP range at the CDN edge."

"Googlebot and Bingbot are on a verified crawler allowlist that bypasses CDN security checks. GPTBot is not currently on that list."

"We cannot commit to a timeline or outcome."

Correction, 15 September 2026

This page originally said the mechanism was "not rate limiting" and described it as a verified-crawler allowlist with the largest AI crawler missing from it. That came from the first agent's explanation, and Hostinger's final answer supersedes it. On 15 September they closed the ticket and stated that their servers "apply a rate limit specifically to OpenAI's training crawler (GPTBot/1.x) at the infrastructure level", that the rule exists to "protect shared server resources for all customers on the platform", and that it is "not configurable per domain".

Every measurement on this page still stands. What changed is the label. A rate limit set low enough that 15 consecutive cache-busted requests all fail is indistinguishable from a blanket block when you are testing it from outside. Slowing down only partly lifts it: at one request every 60 seconds on 2026-09-21, 2 of 6 previously refused URLs came back. We could not tell the difference from the client side, and we should have said so rather than ruling one out.

"The 429 responses for GPTBot are intentional and server-wide, this is not caused by anything on your website or your hPanel configuration."

"A different User-Agent string returning 200, it simply doesn't match the rate-limit rule; it does not mean your site behaves differently for that crawler."

"The issue appearing on multiple domains, because the rule is server-wide, not domain-specific."

"While we are unable to allowlist GPTBot on shared/cloud hosting, here are your available options."

"We're sorry we can't offer a per-domain exception on the current hosting plan."

The three options they offered: raise the cache TTL so more requests hit a warm cache, upgrade to a VPS or dedicated server where "you have full control over server-level rules, including the ability to remove or adjust crawler rate limits entirely", or do nothing, on the grounds that OAI-SearchBot and ChatGPT-User already reach the site. Their own words on that last point: "Your site is not invisible to ChatGPT", which matches what we measured in section 02.

So the finding is narrower and more deliberate than we first described, and in one way worse. It is not an oversight in a list. A major shared host has chosen to rate-limit OpenAI's training crawler across its platform, will not make exceptions, and prices the remedy as an upgrade. Anyone publishing on shared hosting is opted into that by default and has no way to see it.

Cache hits pass, so the only customer-side lever is making more requests hit. Two changes did that: the CDN TTL went from ten minutes to four hours, and Vary: User-Agent was removed from the response, because it had been giving every distinct crawler string its own cache key. GPTBot could never warm its own key, so it was locked out of a cache other visitors had already filled.

What it does not fix is the thing that matters most to a publisher. A newly published page has no cached copy, so GPTBot's first attempt is always a miss, and always 429. Hostinger said the same in the ticket: the cache changes are a workaround, and every new cache miss still returns 429. New content stays invisible to the training crawler until the rule is removed, which is the opposite of what a site that publishes for a living needs.

Five minutes, one terminal, no tool. The method has three rules: defeat the cache with a random query string on every request, send the real crawler's full user-agent string rather than the bare name, and run the controls from the same machine within the same minute so nothing but the identity differs.

# 1. GPTBot, full string, cache-busted
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  "https://YOURSITE.com/?miss=$RANDOM" | head -n 1

# 2. Control: an ordinary browser, same second
curl -sI -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0 Safari/537.36" \
  "https://YOURSITE.com/?miss=$RANDOM" | head -n 1

# 3. Control: OpenAI's other crawler
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" \
  "https://YOURSITE.com/?miss=$RANDOM" | head -n 1

# 4. Control: Googlebot
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  "https://YOURSITE.com/?miss=$RANDOM" | head -n 1

Read the result in this order. If all four return 200, you are fine. If GPTBot alone returns 429 or 403, look at the response headers: a Server value naming the CDN and a request ID with no cache status means the edge refused it before your site was involved. Repeat GPTBot ten times to rule out a budget. Then run the same four commands against a second site on the same host, ideally one you did not build. If it fails identically, you have the evidence a support agent needs, and the words to use are "reproduced on two unrelated domains, controls pass from the same IP in the same minute."

Two traps. A 200 on a page you visited a minute ago proves nothing, because the CDN served the copy it already had; the random query string is not optional. And a short sequence of 200s from a cold start proves nothing either; we saw first requests pass and then everything fail. Ten cache-busted requests is the minimum for a claim.

The standalone version of this method, written for any host rather than for this one case, is how to check whether AI crawlers can reach your site.

The raw run files behind this page are published alongside it: the 64-request run, the header capture, the log split and the 2026-09-21 re-test, per URL with cache status. CC BY 4.0.

This was diagnosed with Claude Code driving the terminal, and it is worth being precise about what that bought. It did not fix anything, because the fix is a rule on someone else's infrastructure. The rate limit sits on Hostinger's edge, and nothing done from this side removes it.

What it did was run the elimination sequence above as scripts rather than as guesses, capture headers on every request so the CDN request IDs existed when support asked for them, and write the ticket from the measurements instead of from memory.

It also produced both wrong turns. The first attribution to our own logging script was a plausible theory that a script wrote up confidently and that a month of evidence then disproved. The lesson there has nothing to do with the tool: a theory about a 429 is worth nothing until the same request has been sent under two identities in the same second, and no amount of reasoning substitutes for that one measurement.

Measured

2026-09-14 10:59:13 UTC

Client

one IP

Requests

64 across 4 paths, all cache-busted

Identities

8, full user-agent strings as sent, listed above

Edge nodes

edge10, edge6, edge7, edge8, edge9

Ticket

#21396332, quoted verbatim

• Single vantage point. Everything here was measured from one IP. The UA-scoped result, with seven identities passing from the same IP in the same minutes, makes an IP-reputation explanation very unlikely, and Hostinger's own diagnostic showed the IP clean, but the test has not been run from OpenAI's published crawler ranges.
• The production log undercounts. After the cache change, hits never reach origin, so the server-side log undercounts every crawler equally. The zero for GPTBot is against 221 for OpenAI's other bots in the same log, so the comparison holds; the absolute numbers do not.
• The bare-word result is reported, not explained. We do not know the exact pattern the rule matches, only that the string OpenAI's crawler actually sends is caught.
• This is one host. Whether other hosts' edges behave the same way is exactly what the method in section 07 is for.

Written by Abdul Subkhan. Published 2026-09-14. Our measurement rules are on the methodology page. Figures regenerate from the run files, so the page and the data cannot disagree.