The question that started this
Around half-way through the stream, a viewer asked Nathan Ingram, our agency trainer, about this most recent update from Cloudflare.
Cloudflare announced updates to Content Independence Day that keeps bots from scraping sites, perhaps by default. Are you making any changes to your settings, or do you keep bot controls off routinely? It takes effect on September 15th.
Shortly after, another viewer made an important clarification: the update will try to block crawlers that don’t clearly distinguish their intent. This distinction matters a lot because it turns this update from a seemingly bad one to potentially very good.
What is “Content Independence Day”?
Starting September 15, 2026, Cloudflare will split AI crawler traffic into three categories, instead of treating “AI bots” as one broad group.
As you can imagine, this allows managing access for those bots to your websites to be a lot more granular. Instead of your rules applying to all bots, you can pick and choose which ones you want and don’t want. Here are the three categories:
Search crawlers: These bots proactively collect and index content so it can be used to answer questions and search queries later. These are the traditional search bots as well as those used by AI models. The expectation is that a site will get something in return for being indexed, like referral traffic or something else of value.
Training crawlers: Crawlers like these pull content specifically to train or fine-tune a model. Unlike the other two, this content is permanently absorbed into the model itself, rather than used to serve a one-off answer or task.
Starting September 15, 2026, any new domain connected to Cloudflare or any new site added to an existing account, as well as any site on the free tier, will be switched to the new rules. All Agent and Training crawlers will be blocked by default on pages that display ads while Search crawlers stay allowed by default.
Of course, if you already have a site on Cloudflare and don’t want anything to change in regard to AI crawler rules, you can opt out through your Security settings any time prior to September 15.
The part that matters more than “AI bots blocked by default”
There’s one detail that’s easy to miss in the coverage of this update: this isn’t a blanket “block all AI” change. Search crawlers are still allowed by default. The real complication happens when crawlers do more than one task.
Googlebot, Applebot, and Bing’s crawler serve more than one purpose. They index web pages for search but they also help AI products . The Search crawlers category refers to crawling that's used to build a search index, regardless of who operates it. That’s fine in Cloudflare’s eyes: it will classify them by whichever purpose they fulfill. Googlebot will likely fall into the Search and Training categories. The issue is that Cloudflare will then apply the strictest rule for those categories.
In other words, under Cloudflare’s new rules, a crawler doing both search and training gets blocked because that’s the more restrictive rule. This is the part worth understanding before you make any changes. Blocking Training crawlers could have unexpected consequences for your site’s discoverability in the places your audience is increasingly searching.
Website visibility and the increased use of AI-driven search
Many people will hear “block AI crawlers” and associate that with protecting their content from being scraped for someone else’s model. We all know how real a concern that is, especially for people who earn a living from their content.
There’s another side to this, though. AI-powered search is becoming one of the main ways people discover websites in the first place instead of the traditional list of blue links (Google search results).
This shift in user behavior changes things. Opting your website out of AI training can have the secondary effect of opting it out of being found by AI tools. With more people using those same AI tools for their searches, you can see where the issue arises. Nathan’s firm opinion when talking about this during the livestream confirms this:
I think it's a terrible idea to block AI bots from your site.
So, the real question now becomes “What does my site actually need protection from, and is blocking the right tool for that?”
Where Cloudflare’s AI crawler blocking makes the most sense is if your site’s revenue depends directly on human pageviews. Say you are an ad-supported publisher: every AI-summarized version of your article is a visit you don’t get, and that’s ad revenue lost. Nathan drew a very clear line when talking about this:
The only time I would block AI bots from a site is if you're legitimately making money from the content that's there. Not if the content is part of a funnel that sells something else, but if you're actually needing people to visit the page for ad revenue in order to make money. That's when I would block bots.
Outside of this scenario, blocking Agent and Training crawlers might cost you discoverability. A portfolio site, service business, or a blog supporting a product usually doesn’t rely on ads to earn revenue, so being discovered is more important. Blocking bots there means you are stopping one of the main ways people can discover the site in 2026.
What to do before September 15
To round out this blog, we’ve put together quick list of actions you can take to prepare for the upcoming changes.
Opt out of the new defaults: If you’d like to do that, you can do so before September, 15, 2026. Simply change the configuration to what you want it to be and it’ll stay that way.
None of this is set in stone so if you notice issues, you can change the settings whenever you want.
Where this leaves you
Circling back to our viewer’s question, this isn’t a case of Cloudflare quietly flipping a switch on every existing website. If you are already using Cloudflare and happy with your setup, nothing needs to change, unless you want it to.
A thing worth noting, though, is that this update is a part of a bigger direction Cloudflare is heading. It aims to eventually enable publishers to get paid by AI companies for access beyond just blocking or allowing their crawlers. You can read more about that here, but it’s not something you must act on today.
For now, Nathan’s main point is what’s worth remembering: for most sites, being found matters more than being protected from something that never costs you anything. And if you have any questions about AI discoverability or crawlers, web hosting, or WordPress, join us for Office Hours every Thursday at 2 p.m. EST where you can get answers live.
FAQs
Does this affect my site if I'm not on Cloudflare?
No. These defaults only apply to sites using Cloudflare's network. If your site sits behind a different CDN or security layer, this specific change doesn't touch you, though it's worth checking whether your provider has (or plans) something similar.
I saw a table on Cloudflare's blog with categories like Transact, SEO, and Ads Verification. Do I need to configure those too?
No. Those additional categories are part of BotBase, a new bot visibility and management layer that's currently only available to Enterprise Bot Management customers. Search, Agent, and Training are the only three categories every Cloudflare customer, including free tier, can actually configure. The rest is internal classification detail, not something you're missing out on.
What happens if I do nothing before September 15?
It depends on your site. If you're setting up a brand-new domain on Cloudflare, or you're on the free tier, the new defaults apply automatically: Training and Agent crawlers get blocked on any page that displays ads, while Search stays allowed. If you already have an existing site with settings you've configured yourself, nothing changes unless you've left things on Cloudflare's defaults.
Will this hurt my SEO?
Not from the Search category itself, since that stays allowed by default. The real risk is indirect: if you block Training crawlers and a crawler like Google bot is bundled as both search and training under one identity, it can get swept up in that block too. It's worth checking your settings specifically for that overlap rather than assuming "I didn't touch Search, so I'm fine."
Is there a reason to block Training or Agent crawlers on purpose?
Yes, if your site's revenue comes directly from human page views and ad impressions, like a publisher who depends on people actually landing on the page. For most service-based businesses, marketing sites, and blogs, that tradeoff doesn't apply, and the visibility benefit of staying open usually outweighs the theoretical downside of AI tools reading your content.