Cloudflare announced a new capability that lets site owners allow search engine crawlers to index their content while blocking AI training bots from scraping the same pages. The feature uses updated robots.txt directives or Cloudflare's bot management to differentiate between legitimate search crawlers and generative AI scrapers. This aims to maintain discoverability in search results without contributing data to AI model training.