Researchers have introduced Taste-Bench, a novel benchmark designed to quantify "taste"—the ability of LLM agents to make optimal decisions during long-horizon engineering and research tasks. Unlike traditional end-to-end benchmarks, Taste-Bench focuses on "decision forks," which are specific points in a trajectory where multiple paths are available and only one leads to a superior outcome. These forks are automatically extracted from parallel task attempts or detours within single trajectories, eliminating the need for human annotation. Evaluation of frontier models reveals a significant performance gap, with the top-performing model correctly navigating only 59.7% of the decision forks. The study finds that task difficulty increases significantly when the evidence required to make the correct choice only becomes apparent much later in the trajectory, and notably, increasing the reasoning budget does not improve accuracy on these decisions. To address these limitations, the authors demonstrate that taste can be enhanced through distillation; by training a student model to mimic a teacher that has access to the full outcome of a trajectory, the student shows improved decision-making on unseen tasks and achieves higher end-to-end success rates on held-out SWE-bench Pro tasks.
Read original
huggingface/daily-papers