One AI newsletter scraper took half of his site's traffic

Posting a side project to Hacker News sends real readers, and then a different kind of visitor turns up and does not leave. Sammy Straus wrote this on his blog in a post that reached 339 points on Hacker News on 13 October: a month after he submitted his customisable feed reader sprinklz.io to Hacker News and picked up about 100 upvotes, his site was still painfully slow, and the slowness kept getting worse several days after the post went live.

What his logs showed

Straus added logging for every request path and found that most requests on the site were not even trying to fetch news articles. His site is a single-page application, so a request that never runs the JavaScript needed to render the page is not a browser belonging to a reader. Looking at aggregate NGINX logs, he found that nearly half of all traffic was coming from one specific user. He describes the user, without naming it, as a service that listens for changes to a website and summarises them into an AI newsletter. "Listens to changes" meant requesting the site many times a minute. He blocked it.

The crawlers that identified themselves, and the ones that did not

The rest of the surge split into two groups. Several of the biggest crawlers were polite and sent recognisable user-agents: Claude's bot, OpenAI's bot and Google's search indexer, among others. Straus wrote a robots.txt to discourage the AI scrapers and keep Google's search crawler, and he added rate limits. He also noted that a share of the traffic was vulnerability scanners probing for files like wp-content/phpinfo.php, on a site that is not WordPress. He asked Claude to walk through the probed paths and confirm no files were leaking, which is a sensible second check for any developer in the same spot.

What a working developer should do with this

Three things, from Straus's account. First, every project that lands on Hacker News will attract a mix of real readers, polite search and training crawlers, and at least one aggressive summariser hitting many times a minute. The quiet slowdown "several days after" the post is the sign to look for. Second, aggregate NGINX logs by user-agent and by request path, because the one abuser is visible when the usual metrics are not: Straus caught the newsletter scraper by noticing that most requests never ran his JavaScript. Third, a plain robots.txt, rate limits and the right crawler policy at the edge handle most of this, and are cheaper than any infrastructure change.

The post also makes a different argument worth noting: reaching the Hacker News front page is now a test of whether a site can hold up under heavy load, as much as it is a gain in readers. Straus wrote that his real lesson is that the site has to get faster, and that he has rebuilt the underlying infrastructure since the incident. On an older architecture, a scraper hitting many times a minute is a denial-of-service that costs the project its real audience.

Source

I posted to HN, and then my website was flooded with AI bot traffic, Sammy Straus.