University libraries across North America, Europe, and Oceania are experiencing severe disruption from AI data-scraping bots that ignore robots.txt protocols and overwhelm servers with thousands of requests per second, crashing catalogues and blocking legitimate users. San Jose State University and the University of North Carolina at Chapel Hill reported incidents where bot traffic — including over 11,000 searches for Finnish music from thousands of IPs in a single day — periodically shut out students and faculty. Unlike search engine crawlers that respect access rules, these bots, deployed by major AI firms like Anthropic, OpenAI, Google, and Microsoft as well as third-party scrapers, download content in every available format without rate limiting. Project Gutenberg's Eric Hellman notes the bots are "too stupid" to use the organization's one-click bulk download of public domain works, instead hammering live sites. Libraries face a dilemma: CAPTCHAs and login requirements deter bots but also hinder human access and beneficial crawlers like the Internet Archive. While major AI companies recently endorsed a voluntary, standardized opt-out signal in robots.txt, compliance excludes smaller startups and third-party operators. Experts warn this extractive behavior creates a disincentive for institutions to maintain open digital collections, threatening a "huge benefit" to public knowledge access built over two decades of digitization efforts.
Read original
hackernews