Yes, Cloudflare's AI bot settings can block Googlebot. Cloudflare says that if you block "Training" crawlers, mixed-purpose crawlers such as Googlebot, Bingbot and Applebot are treated by their strictest behaviour and blocked too. The new defaults were scheduled for 15 September 2026, so as of 9 October 2026 the fix is simple: open your Cloudflare AI bot settings, confirm Search crawlers are allowed, and test a live URL in Search Console.
Key Takeaways
- Cloudflare's 1 July 2026 announcement split crawlers into Search, Agent and Training, with new defaults from 15 September 2026.
- Cloudflare states that crawlers doing both search and training are judged on all their behaviours, so a Training block can also block Googlebot.
- New domains and unchanged free-plan sites were the ones most exposed to the default change; sites that already enabled "Block AI bots" were also flagged by coverage.
- One Reddit-sourced report (via Search Engine Journal, August 2026) described 403 errors for Googlebot after enabling an AI training block. It was anecdotal and unconfirmed.
- Check this week: Cloudflare settings, a Googlebot-verified log sample, and Search Console's live URL test.
- I could not find a Google statement or a confirmed wide-scale deindexing case as of 9 October 2026, so treat this as a risk to rule out, not a disaster in progress.
What Cloudflare actually announced
Cloudflare's own post, published 1 July 2026, says site owners can now manage Search, Agent and Training bots separately, including on the free tier. The effective date for the new defaults is 15 September 2026. For new domains, Training and Agent bots are blocked by default on pages that display ads, while Search stays allowed (Cloudflare blog).
The sentence that matters for SEOs is about mixed crawlers. Cloudflare says they are allowed or blocked "according to all of their behaviors", and names Googlebot, Applebot and Bingbot as ones that would be blocked for customers who block Training. Search Engine Journal's write-up reports the same rule and adds that "Verified" bots no longer get automatic access everywhere (Search Engine Journal).
Why Googlebot is the awkward case
Googlebot is one crawler that feeds ordinary Search, AI Overviews and AI Mode. There is no separate crawler you can allow for results and block for AI features, and the robots.txt control Google-Extended does not cover Search features. A write-up of the change makes the same point: blocking Googlebot at the network level also stops it fetching pages for normal results (hobo-web summary).
Cloudflare's stated intent is to pressure operators into running separate crawlers for search, agents and training. That is a policy fight between a CDN and Google. Your site just needs to stay out of the crossfire.
Who is most exposed
Based on Cloudflare's post and the coverage I opened:
- New domains onboarded to Cloudflare.
- New sites added by existing customers.
- Free-plan customers who never touched the settings before 15 September.
- Anyone who previously ticked "Block AI bots" or enabled an AI Training block.
If you are a paying customer with custom bot rules, the exposure depends on your configuration. I could not verify from Cloudflare's documentation exactly how every plan was migrated, so check, don't assume.
What the evidence says so far
In August, Search Engine Journal reported a Reddit post where a site owner with "AI Training" set to Block saw Googlebot and Bingbot get HTTP 403 on a sitemap fetch. Disabling the block restored access, and Bot Fight Mode produced the same 403s in a follow-up. John Mueller asked the poster to send details privately. The article itself says it was unclear whether it was a fluke or user error (Search Engine Journal).
Several agencies and tool vendors published check guides after that, for example Playwire and Ipregistry, which discusses why verifying crawler IPs is now part of the picture. I have only seen their search summaries, not audited their claims, so I am not using their numbers. One single-author note also says a robots.txt "use=" directive now exists for content-use preferences; Cloudflare says those are preferences, not blocks.
The 10-minute check, step by step
- Open Cloudflare's AI bot policy panel in Security settings. Look for the Search, Agent and Training rows. Confirm Search is Allow.
- Look for legacy toggles. "Block AI bots" and Bot Fight Mode are the two that appeared in the reported 403 case.
- Decide on Training deliberately. If you want to block AI training, you now need to check what that does to mixed crawlers on your plan.
- Test in Search Console. Use URL Inspection, then "Test live URL". Check "Page fetch" says Successful (Search Console Help). Note the test runs as Google-InspectionTool, so a pass is encouraging, not proof Googlebot in production is fine.
- Check Crawl Stats. In Search Console, Settings, then Crawl stats. Look for a jump in 403 or 5xx responses after 15 September.
- Sample your logs or Cloudflare firewall events for genuine Googlebot requests that were challenged or blocked.
Verify it is really Googlebot
Before you whitelist anything, confirm the IP. Google documents a reverse DNS lookup (result should end in googlebot.com, google.com or googleusercontent.com), then a forward lookup that resolves back to the same IP. It also publishes IP range JSON files such as common-crawlers.json (Google verification docs).
This matters because fake "Googlebot" user agents are common. A rule that allows anything claiming to be Googlebot is a hole, and a rule that blocks anything not verified by Cloudflare's own list may be too tight.
If you find a block: fix and recover
- Change the policy so Search crawlers are allowed. Re-test live URLs.
- Keep a dated note of when the block was on. That helps you explain a dip later.
- Do not hammer Googlebot with a flood of re-crawl requests. Fix the cause, then let crawling recover. For the "slow Google down on purpose" side of this, see my breakdown of Retry-After and crawl rate.
- If URLs were returning errors for days, expect delays. Google's crawl documentation warns that URLs seeing persistent error codes over several days can drop from the index (Google crawl rate docs).
The AI-training decision, separated from the SEO decision
Blocking training is a legitimate business choice, particularly for publishers with ad revenue. Just make the two decisions separately.
- SEO decision: Must Googlebot reach every indexable URL? Almost always yes.
- AI-training decision: Do you want other training bots blocked? Use bot-specific rules and robots.txt entries for named crawlers, then re-run the Googlebot test after every change.
If you also care about being cited by assistants, remember that blocking agent crawlers can reduce live retrieval. I covered how citation works in an earlier AI citation definition post.
What I do this week
On client sites I put a recurring check in the weekly review: one live URL test on a template page per section, one Crawl Stats glance, and a note of any CDN or WAF settings changed. Edtech and course sites are especially prone to over-aggressive bot rules because they have fought scrapers. For my own reading list, the Google spam update status is the other thing I check weekly.
Caveats and what I could not verify
- I did not find a Cloudflare or Google statement after 15 September confirming how the rollout went, or whether Googlebot was blocked at scale.
- Plan-specific migration behaviour is not fully documented in what I opened.
- The only incident I found is a single anecdotal report. Treat third-party "X% of sites blocked" claims with suspicion unless they show method.
Verified as of 9 October 2026 against the sources linked; this is operational guidance, not legal or contractual advice.
FAQ
Does Cloudflare block Googlebot by default?
Not for everyone. Cloudflare's default from 15 September 2026 blocks Training and Agent bots on ad-displaying pages for new domains and unchanged settings, and keeps Search allowed. The risk is that a mixed crawler like Googlebot is judged on all behaviours, so a Training block can catch it.
How do I know if my site is affected?
Open your Cloudflare AI bot settings, run a Search Console live URL test, and compare Crawl Stats before and after 15 September. A spike in 403s or fetch failures is the signal.
Is "Block AI bots" the same as the new Training block?
Cloudflare says the new Search, Agent and Training options replace the legacy setting from 15 September. Search Engine Journal reports sites that already enabled it are covered by the mixed-crawler rule.
Will a blocked Googlebot remove my site from Google?
Not instantly. Persistent errors on a URL over several days can lead to it dropping from the index, according to Google's crawl documentation. Fix quickly and monitor.
Can I block AI training and still allow Googlebot?
Possibly, but you have to verify it on your plan. Use named bot rules rather than blanket toggles, then re-test with a live URL inspection.
Does Google-Extended solve this?
No. Google-Extended is a robots.txt control, and Search Engine Watch-derived coverage notes it does not offer a separate opt-out for Search features like AI Overviews and AI Mode.
Is the Reddit 403 report proof of a bug?
No. Search Engine Journal said it was unclear whether it was a fluke or user error, and Mueller asked for details privately. It is a reason to test, not a conclusion.
How do I confirm a request is real Googlebot?
Use reverse DNS then forward DNS, or match the IP to Google's published JSON ranges. Never trust the user-agent string alone.
Should I whitelist Google's IP ranges in Cloudflare?
That can work as a belt-and-braces rule, but keep the list updated and combine it with verification. Test after saving.
Want a second pair of eyes on your crawl access?
I'm Younus Fardeen, an organic growth, SEO and AEO strategist with 4+ years of marketing experience across edtech and startups. If you want a crawl-access and bot-settings audit, or a content plan that survives the next update, see my work and reach me through the contact form at https://younusfardeen.in.