A research lab conducted a sealed benchmark testing whether llms.txt files improve model performance across 8 arms (4 local Q4 and 4 frontier cloud models) using 30 standardized questions about their own website content. The study compared three conditions: providing llms.txt files, providing raw HTML, and providing neither, with results that contradicted their pre-registered hypothesis. Despite the llms.txt movement being two years old with limited empirical evidence, the published findings show minimal to no benefit from llms.txt files across both local and cloud-based models.

Read original