University libraries say AI bots are knocking their archives offline

A public library site that used to serve students and faculty is now sharing bandwidth with a bot farm. CBC News reported on 5 November 2026 that academic libraries are being knocked offline by AI scraping bots. The libraries are in North America, Europe and Oceania. One Canadian expert told CBC the problem has got worse through 2025 and 2026.

Reporter Kevin Maimann quotes Jill Strykowski at San Jose State University. She says the bots are "bombarding us with trying to grab up as much material as possible". The bots feed Claude, Gemini and ChatGPT. Libraries, museums and archives are targets. Their content is vetted and not hidden behind paywalls.

One day, 11,000 searches for Finnish music

CBC gives one measured example. The University of North Carolina at Chapel Hill got more than 11,000 searches for Finnish music in a single day. The traffic came from thousands of different IP addresses. The library catalogue periodically shut out real students, staff and faculty while the bots worked.

Dean Seeman at University of Victoria Libraries told CBC the traffic can be worse. In his words, the sites can "go down completely for a little while". Eric Hellman at Project Gutenberg told CBC the bots can go very fast. They sometimes "request thousands of documents a second and download them in every available format". Hellman said the old fix no longer works. Blocking a single IP is useless when one bot farm runs from thousands of addresses.

What the AI companies say they will do

The CBC piece notes the four largest AI firms moved last week. Anthropic, OpenAI, Google and Microsoft said they would honour one standard opt-out signal in robots.txt. Any of them could use it to block their crawlers from a site. The agreement is voluntary. It also leaves out third-party scrapers and smaller labs. That is where the hardest traffic is coming from.

Some libraries have added CAPTCHA checks or forced login. Michael Weinberg at NYU Law School's GLAM-E project said those fixes have a cost. They also lock out archive bots run by Internet Archive and Project Gutenberg. Libraries want to keep those bots. Hellman told CBC what they want from the training crawlers. "We just wish they would do it in a respectful way that doesn't take down websites."

What a team running a public site should do this week

Any team running a small public site can see the same traffic now. A library catalogue, a documentation site, an open data portal, a jobs board are all targets. A site owner has three things to check this week. Count real bot traffic in the server log, by user agent and by IP range. Add a rate limit per IP range, not per single address. Watch for the shape in the CBC story. Thousands of requests from thousands of addresses, all reading the same section, inside one day.

The policy work by Anthropic, OpenAI, Google and Microsoft is useful but not enough. The three fixes above do not wait on any crawler to behave.

Source

• University libraries are being swarmed by AI bots, putting public access at risk, CBC News (Kevin Maimann)