Researchers have introduced OmegaUse-OfficeVal, a novel benchmark designed to evaluate Large Language Model (LLM) agents performing long-horizon office-suite workflows through the lens of economic grounding. The dataset consists of 100 complex tasks adapted from real-world practitioner requests via a privacy-preserving methodology. To facilitate value-weighted evaluation, each task is paired with two specific economic signals: an estimate of human labor time—averaging 2.32 hours per task—and a task price proxy. This framework allows for a direct quantitative comparison between the cost of LLM inference and the cost of human labor. To ensure objective assessment, the authors implemented code-based verifiers derived from fine-grained rubrics. Experimental evaluations involving several frontier LLMs and a human baseline reveal a significant performance gap; while the tested LLMs demonstrated superior speed and lower operational costs compared to human workers, they failed to achieve human-level quality in task deliverables. By integrating economic metrics with complex task execution, OmegaUse-OfficeVal provides a standardized method for determining the practical utility and cost-effectiveness of autonomous agents in professional office environments.

Read original