Researchers from Xiaohongshu Inc have introduced VibeLifeBench, a novel benchmark designed to evaluate the proactivity and persistence of Large Language Model (LLM) agents within complex, long-horizon lifelike scenarios. Unlike traditional benchmarks that focus on short-term, static requests, VibeLifeBench consists of 200 tasks spanning ten everyday-life domains, designed to simulate multi-week timelines. The evaluation environment features a simulated world containing 22 mock services that advance according to an autonomous clock, introducing "silent changes" that require agents to actively re-inspect the environment to detect updates. The scoring methodology utilizes fine-grained, weighted checks that assess the agent's final state, the timeliness of its interventions, and its ability to adhere to implicit constraints. Upon evaluating seven frontier models, the study found that all tested models scored poorly, highlighting a significant gap between current LLM capabilities and the requirements for reliable, autonomous life assistance in dynamic, evolving environments. The researchers intend to open-source the task sets, environments, and the evaluation framework to facilitate further development in the field of proactive agentic behavior.
Read original
huggingface/daily-papers