Installing an AI crawler tracking plugin is a good idea. It can give you visibility into things you might not otherwise notice. A viewer during the July 30, 2026 Office Hours did, indeed, notice something they had been missing so far.
The plugin told him that his website, and specifically the author page, is getting bombarded by web crawler traffic. It was Amazon’s bot, the one that trains its AI products and Alexa. Rather than panic, he did something smart, which is what inspired this post.
You’ll learn about how our viewer seized the situation to his benefit, what Nathan Ingram, our agency trainer here at hosting.com thinks, and when’s a good time to block AI bot traffic.
The viewer’s situation
Around the half-way point of the livestream, a viewer brought up something very interesting that’s been happening to their website for a while. Here it is in their words:
I noticed the Amazon bot is crawling my website like crazy. In less than a week, 167,000 requests. What's interesting is that my author page has crawled by far more than any other page. I went and optimized my author page by adding a description and social links.
I started a debate on Reddit about whether I should block the Amazon bot. It's a training bot for AI service, product, and Alexa. The recommendation I've gotten is I should not block the bot unless I feel my website response is suffering.
An SEO expert also told me, “Remember, you want to be in the training data.”
The first thing we are sure you’ll notice, because it caught Nathan’s attention too: the 167,000 requests in less than a week. That’s a startling amount but, as you can see from the quote, it isn’t necessarily overwhelming.
And our viewer smartly took advantage of the situation. The bot is obviously expending a lot of resources to keep returning to the page, so by improving it he increased his chances of getting cited in AI searches. That’s great for his site’s SEO.
But there’s one more side to this: don’t block the bot unless the website’s response is suffering. That’s what we want to further unpack here.
Should you block AI crawlers?
Fortunately, that question has a clear answer: in general, no. Blocking an AI crawler means opting out of any chance of your content appearing in or getting cited by the AI tools people are increasingly using.
That’s why our viewer received the advice they did: unless the crawler is actively harming your site’s performance or health, blocking it is usually worse than leaving it alone. Of course, that doesn’t mean you shouldn’t keep an eye on crawler traffic.
Even if the traffic isn’t hurting your site, it could tell you other things about it that you might not have yet considered. Nathan explained it well:
Can we say charitably that that's excessive? That's a lot. That would be borderline of me wanting to do something about it. But it depends on if you're on metered hosting or not.
On one hand, you can benefit from it, as our viewer did. On the other, though, you might want to check if the volume is costing you anything. Even if it’s not impacting your server’s performance, it could be increasing your bill if you are on metered hosting.
So, in general, you want to leave the bots alone. It’s also the advice our viewer got from an SEO expert. Leaving them alone increases your chances of getting discovered.
When it makes sense to block AI crawlers
There are occasions when blocking crawlers is the better choice, though. We have already mentioned two of them, as those are the ones Nathan and our viewer brought up, but there’s one more that’s perhaps even more important. Let’s elaborate on all three of them:
Metered hosting: Metered hosting means your hosting plan counts bandwidth or requests against a cap. Once you go beyond that cap, you incur additional fees. As AI crawler traffic uses up both bandwidth and requests, it’s worth evaluating if the visibility is worth the potential additional cost, or if it’s time for a hosting plan built for this kind of traffic.
Performance is suffering: Site performance can also degrade from immense crawler traffic. Each time a crawler requests your website, that elicits a response from the server. If there are too many of these crawler requests, it could overwhelm the server, harming genuine human customer experience.
Ad revenue: AI crawler traffic can also impact your ad revenue. Every time an AI search surfaces your content, that’s one visitor who doesn’t go to your website. If you are relying on ad revenue to monetize your website, that’s a real downside that warrants blocking AI crawlers.
Outside of those scenarios, opting out just means you are giving up a visibility channel that’s only going to matter more as AI becomes even more prevalent.
It’s worth noting that on September 15, 2026, Cloudflare is rolling out new default settings for AI crawler access which you should check out if you are using their services (even the free tier). These new defaults are strict and specifically target pages that display ads, blocking Training and Agent crawlers there unless you opt out. So, if ad revenue is a concern for you, this update is especially worth a look before the deadline.
The bigger opportunity: give crawlers something worth finding
The most useful takeaway from all of this isn’t about blocking or allowing crawlers. That’s a purely situational choice dictated by the three factors described above. Instead, pay attention where on your website they are already spending their time.
If an AI crawler tool shows you that one or several pages are getting a disproportionate amount of attention, that’s a signal. It means the bots see something valuable on those pages and are weighing it more heavily when forming an opinion about your site.
Here’s a quick, practical list you can go through when you notice a pattern like this:
Check the traffic: An AI crawler visibility tool will show you which bots are hitting which pages. That eliminates you having to guess and gives you concrete information.
Look for the outliers: Find the pages the bots seem most interested in. Ask yourself why that is and determine if the page is worth the attention it’s getting. The bots aren’t perfect, and they could sometimes be staring at a page that’s not that important.
Fill in the gaps: After determining the importance of the page, make it better. In our viewer’s case it was a thin author bio and missing social links that they had to add. In your case, it might be something different, so give the bot the best version of that page if it will keep coming to it.
Don’t confuse volume with problem: High crawler traffic isn’t a problem on its own. In reality, it’s arguably a healthy sign that your content is on the AI’s radar.
Don’t treat pages bots are interested in as an afterthought. Do as our viewer did and use this opportunity to benefit from the attention. AI crawlers are only going to get more sophisticated which, we hope, will lead to website owners putting more effort towards their content.
AI bot traffic isn’t just noise anymore
Rather than treating AI crawlers as something to block, as the tendency might have been a couple of years ago, treat them as signals. They will show you which parts of your website are earning attention.
Make sure those parts are worth finding and worth your visitors’ time. If you are wondering if there’s anything else you can do to show up well for AI bots, you can check out our “Is your site agent ready?” livestream.
Finally, if you have any additional questions about AI crawlers, AI in general, WordPress, or web hosting, join us every Thursday at 2 p.m. ET for Office Hours. Nathan Ingram and the Agency Success community talk about all sorts of things, and you are always bound to learn something new.
FAQ
Is AI crawler traffic different from regular search engine crawler traffic?
Yes. Traditional search crawlers like Googlebot index your content so it can appear in search results. AI crawlers can serve different purposes: some train AI models, some power AI search or agent tools, and some do a mix of both. That distinction matters, since blocking one doesn't necessarily block the other. Increasingly, tools like Cloudflare are asking site owners to manage each category separately.
How do I know which bots are crawling my site?
Most hosting environments log this at the server level, but a dedicated AI crawler visibility plugin makes it far easier to see which bots are hitting your site, how often, and which pages they favor, without digging through raw server logs.
Will blocking AI crawlers hurt my ranking in regular search engines like Google?
Not if you're careful about which bots you block. Search-focused crawlers and AI training/agent crawlers are increasingly treated as separate categories by tools like Cloudflare, so you can typically restrict AI training bots without touching your regular search visibility. The risk is in blunt, all-or-nothing blocking that catches search crawlers too.
My site doesn't run ads. Does the ad revenue concern even apply to me?
No, that particular downside is specific to ad-monetized sites. If you're not running ads, the more relevant factors are metered hosting costs and server performance, which are covered above.