In this edition of Office Hours, we tackle one of the most important questions agencies are asking in an age of AI-powered discovery: How do I make sure that large language models (LLMs) like Claude, ChatGPT, and other AI content agents can actually access and “crawl” my website when it’s behind Cloudflare or hosted through hosting.com’s MWP platform powered by Rocket.net?
This question came up during a recent session with Nathan Ingram during hosting.com Agency Success Office Hours, where a participant was blocked from testing URL redirects with Claude because it couldn’t crawl the site. Nathan walked through troubleshooting steps and practical insights. And in this blog, we build on that with deeper context, clearer workflows, and additional credible background on how Cloudflare and AI crawlers actually behave.
Why AI crawler access matters
Traditionally, the concept of crawling was tied to search engines like Google and Bing. These search engines send bots to your site, read HTML content, metadata like titles and descriptions, and index that content so users can discover it in search results.
But today, AI systems also rely on this crawling process - not just to index pages for search, but to ingest content to generate answers, summaries, or training data. Tools like Claude, ChatGPT, Perplexity, and other AI assistants increasingly answer questions by pulling from web content, either in real time or from previously indexed material. If these systems can’t access a page, they can’t reference it or quote it in their responses, which means your content vanishes from an increasingly prominent set of discovery channels.
This creates three major practical shifts for agencies and content owners:
AI Search Visibility - AI answers, summaries, and references are becoming part of how people discover brands, services, and expertise. Being excluded from that layer limits your reach even if your SEO is strong. AI tools like Claude or ChatGPT can drive awareness and referral traffic, but only if they can read your content.
Brand Authority - When AI tools cite or reference your domain, it reinforces credibility. When they cannot access your content, your competitors may fill that gap instead. Being cited directly by intelligent assistants boosts trust and domain visibility.
Future Monetization - Some platforms are experimenting with structured ways to monetize AI access to content. Cloudflare and other platforms are experimenting with models where content access for AI could be monitored or monetized. Whether you opt in or out of that conversation depends on how you configure your site today.
The practical reality is this: AI visibility is becoming a new form of discoverability. You need to decide whether you want in or out.
Cloudflare’s shift in AI crawler controls
Cloudflare is one of the most widely used edge networks and security layers on the web. Historically, websites were “crawlable” by default. Cloudflare primarily focused on blocking malicious bots, scrapers, or abusive traffic, rather than AI specifically. But for content owners, AI crawling has grown into a challenge - chatbots and generative AI systems scrape huge volumes of data without sending meaningful traffic back to the original content source.
Over the last couple of years, that posture has changed.
Cloudflare now treats AI crawlers as a distinct category of traffic. Instead of assuming they are welcome, Cloudflare gives site owners explicit control over whether AI systems can access their content.
AI crawl control
Cloudflare now provides tools under AI Crawl Control that let you explicitly allow or block specific AI crawlers. With these, you can:
See AI crawler activity and how frequently bots request content
Choose to allow or block each AI crawler at the network edge
Optionally include monetization through pay-per-crawl (currently in beta)
This approach means your site no longer assumes AI bots are welcome by default. You make the choice.
Default blocking on new sites
In a major shift in 2025, Cloudflare began blocking AI crawlers by default on new domains unless you actively enable access. This flips an old assumption. Previously, if you did nothing, bots could usually crawl. Now, if you do nothing, some AI tools may be blocked before they even reach your server; instead of AI accessibility being automatic, it’s opt-in.
That makes configuration essential; if you want LLMs and AI tools to read your content, you need to actively permit them.
Step-by-step: Allowing LLM crawling on Cloudflare
Now that we have established how vital it is for crawlers to have access to your site’s contents, here’s a practical checklist to make sure your site is crawlable by AI tools like Claude.
1. Confirm your Cloudflare AI Crawler settings
Log in to the Cloudflare Dashboard and go to your domain’s settings.
Look for:
AI Crawl Control - This is where you can explicitly allow specific crawlers by name or category.
Bot traffic settings - Cloudflare has toggles like “Block AI bots” that may be active by default. For AI visibility, you may want this inactive so that API-based crawlers aren’t blocked unwittingly.
Because Cloudflare can block AI crawlers at the network layer, your first task is to ensure these aren’t set to “block”.
2. Review your robots.txt file
A robots.txt file at the root of your site communicates crawl directives to bots. Traditionally, it’s been used for search engine crawlers, but modern AI bots generally respect this file too.
To allow AI crawlers, make sure:
The sections you want indexed do not have disallow rules for known AI user agents.
You have a valid sitemap reference, so crawlers know where your main pages are.
Robots.txt might look like this to allow general crawling:





.webp)