The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see.
A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.
And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.
Over the last 72 hours, Claude-searchbot [1] alone fetched ~205,000 pages. Sent exactly 1 referral. There is a lot of free financial data on the site, hoping for real users to benefit from it. It is hard to not feel a little cheated out that Claude gets to claim "Found it!" to its users without me getting no credits or compensation whatsoever.
[1] Exact user agent `Claude-SearchBot/1.0; +searchbot@anthropic.com)`
> My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%.
This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.
tarr11
Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software.
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
qbane
Can someone help me understand the underlying motivation behind this?
It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely.
What's the motivation behind the same entity re-fetching a page it just fetched less than an hour ago?
varenc
I'm in a similar boat. Probably 99.999% is bots. I have nearly 1m unique "visitors" according to cloudflare and my real users are in the dozens a day. That being said, I love the open internet and am holding on to keeping as much open as I can.
ddxv
This will resonate with anyone who operates a public-facing website and doesn't work at a big tech company. In the past few years I have also been down this exact same road. I'm desperately trying to avoid resorting to Cloudflare, but I'm running out of time and patience to keep tweaking nginx and firewall rules every few weeks.
I had initial success with https://git.gammaspectra.live/git/go-away as a more powerful and more reasonable self-hosted alternative to Cloudflare than Anubis, but it seems to have gone unmaintained. There is also https://github.com/dgl/haphash if you run HAProxy, though I have not tried it.
On that note, does anyone know what happened to Ted Unangst aka tedu? He was a prolific OpenBSD developer and blogger, and he had developed one of his own simple solutions https://humungus.tedunangst.com/r/anticrawl, but all his web properties seem to have gone away recently...?
aorth
I run scripts on my servers on an hourly basis to check which are the top 25 IPs visiting the server (aggregated by /24). If anyone in those top 25 IPs are from China, Vietnam, etc, or from Alibaba/Amazon/etc, the /24 gets blocked by iptables.
It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam.
However, on a Gitlab instance I manage (500 users), we have to restrict viewing of git logs and pretty much everything except issues. The bots were too aggressive. Chinese crawlers have access to a huge range of IPs and they often do only 10-20 requests per day, while generating in total over 50k requests per day. Our server load went from 99% down to 0.1% after that (and it's a fairly big server).
mgbmtl
I made a small booking site for a local dutch canal boat, which has a calender function. A simple PHP app. I checked the Apache logs recently, and it had THOUSANDS of claudebot and other AI UserAgents flooding the logs every day, apparently because the scrapers keep hitting the 'next month' button on the calendar in a an infinite loop, all day, everyday!
This is a small booking app without any useful information at all, it surprises me that the AI boots have no discernment about what the are scraping, just wasting their own and other peoples resources. And their own reputation! You would thing they could spare a few tokens on a classifier model to do a quick evaluation of their scraping efforts, but apparently they do not.
Anyway, I have done my best to block these UAs and so far it seems to have improved the situation.
comments (10)
A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.
And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.
jwr
Over the last 72 hours, Claude-searchbot [1] alone fetched ~205,000 pages. Sent exactly 1 referral. There is a lot of free financial data on the site, hoping for real users to benefit from it. It is hard to not feel a little cheated out that Claude gets to claim "Found it!" to its users without me getting no credits or compensation whatsoever.
[1] Exact user agent `Claude-SearchBot/1.0; +searchbot@anthropic.com)`
Proof: https://i.postimg.cc/Pqc3SS8T/Screenshot-2026-08-07-at-5-33-...
GodelNumbering
This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.
tarr11
[1] https://anubis.techaro.lol/
johnorourke
qbane
It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely.
What's the motivation behind the same entity re-fetching a page it just fetched less than an hour ago?
varenc
ddxv
I had initial success with https://git.gammaspectra.live/git/go-away as a more powerful and more reasonable self-hosted alternative to Cloudflare than Anubis, but it seems to have gone unmaintained. There is also https://github.com/dgl/haphash if you run HAProxy, though I have not tried it.
On that note, does anyone know what happened to Ted Unangst aka tedu? He was a prolific OpenBSD developer and blogger, and he had developed one of his own simple solutions https://humungus.tedunangst.com/r/anticrawl, but all his web properties seem to have gone away recently...?
aorth
It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam.
However, on a Gitlab instance I manage (500 users), we have to restrict viewing of git logs and pretty much everything except issues. The bots were too aggressive. Chinese crawlers have access to a huge range of IPs and they often do only 10-20 requests per day, while generating in total over 50k requests per day. Our server load went from 99% down to 0.1% after that (and it's a fairly big server).
mgbmtl
This is a small booking app without any useful information at all, it surprises me that the AI boots have no discernment about what the are scraping, just wasting their own and other peoples resources. And their own reputation! You would thing they could spare a few tokens on a classifier model to do a quick evaluation of their scraping efforts, but apparently they do not.
Anyway, I have done my best to block these UAs and so far it seems to have improved the situation.
rsolva