Cloudflare was serving a robots.txt we did not write

This is one section of the Pakistani developer's guide to domains, hosting and deployment, written up on its own because it cost us months.

Our public/robots.txt is four lines long. It allows everything and points at the sitemap. It has been in the repository since the site launched, it deploys with every build, and it is correct.

The file being served at uxprosolutions.com/robots.txt was 1,908 bytes and it blocked GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, Amazonbot, Applebot-Extended and meta-externalagent, with a Content-Signal: ai-train=no header for good measure. Our four lines were appended underneath, where they did nothing.

What happened: Cloudflare sits in front of our Vercel deployment, and it has a managed robots.txt feature that injects its own AI-crawler blocking rules before the origin file. It was on by default. Nothing in our repository, our build or our Vercel configuration could have changed it, because the file never reached the visitor.

How we found it, which is the useful part

Not by looking at robots.txt. Nobody looks at robots.txt.

We found it running a crawl audit with a Googlebot user agent across every URL in the sitemap, comparing what the crawler receives against what the repository contains. The script fetches each page, extracts titles, canonicals, headings and metadata, and prints anything that does not match expectations. Checking robots.txt was almost an afterthought in it.

The tell was the byte count. Repository file: 72 bytes. Live file: 1,908 bytes.

That gap is the whole lesson. Every other check we ran on that deployment passed. Prerendering worked, canonicals were correct on all seventeen URLs, the structured data validated. The one file that governs whether any of that gets read was being replaced in transit, and nothing in our own stack knew.

What it cost

Hard to quantify, and worth being honest about rather than dramatic.

Google was unaffected. Googlebot was never blocked, so classic search indexing carried on normally. Google-Extended was blocked, which controls whether content can be used for Gemini grounding and AI Overviews, so we were opted out of that.

The larger cost was the assistant crawlers. For a software house in Quetta with no brand recognition outside Balochistan, an AI assistant answering "who builds software in Balochistan" is a distribution channel we do not pay for and cannot win on a conventional SERP against directories. We had been blocking every one of them for months without knowing.

We have no way to measure the traffic that did not happen. Anyone who tells you they can measure that is guessing.

The fix

Cloudflare dashboard, your domain
  Security, Settings, or Security, Bots, depending on your plan
  Find the managed robots.txt or AI crawler blocking rule
  Turn it off

Then verify from outside, because verifying from inside is how the problem survived in the first place:

curl -sS https://yourdomain.com/robots.txt
# PASS: only your origin file
# FAIL: any line containing "Content-Signal" or a managed block header

Ours now returns 72 bytes and nothing else.

If the managed block survives after switching it off at zone level, it is applied at account level instead. Check Account Home, Configurations, and look for a separate AI scraper setting that applies across every domain in the account.

The general problem, which is not really about Cloudflare

A proxy in front of your origin can modify anything it serves. That is what it is for. Most of what it does is helpful: caching, compression, TLS, DDoS absorption. The category of problem this incident belongs to is that a proxy can also change things you consider yours, on a default you did not choose, and your repository will keep telling you everything is fine.

Cloudflare turned this on for a defensible reason. Plenty of site owners genuinely do not want their content in training corpora, and defaulting to protective is a reasonable product decision. Our position is different, and the point is that the position should be ours to take.

Our opinion, since this piece is one long argument for it: at our size, block none of them. There is no paywalled archive to protect and no licensing deal on the table. What there is, is an absence of brand recognition, which every one of those crawlers helps with. Revisit it when you have published something worth licensing and somebody offers to license it. Not before.

What to check on your own site this week

1. curl your live robots.txt and compare it byte for byte with the file in your repository.
2. curl -I your homepage and look at cf-cache-status and Cache-Control. Ours said DYNAMIC and max-age=0, must-revalidate on a completely static document, which is a separate problem in the same family and one that undoes whatever your host's network gains you.
3. Fetch a URL that should not exist and check the status code, not the page. Ours returned 200 with the homepage body, which is worse than the robots.txt issue and took the same audit to find.

All three take ten minutes together. None of them show up in any dashboard.

Talk to us

If you have a proxy in front of your site and have never checked what it serves, that ten minute check is worth doing today. If you would rather someone else ran it across your whole site, we do that.