Researchers have introduced OSReward, a new benchmark designed to evaluate the reliability of Vision-Language Models (VLMs) acting as judges for Computer-Using Agent (CUA) trajectories. As human verification is unscalable, the field relies on VLMs to assess whether an agent's actions and reasoning successfully fulfill instructions. The OSReward framework utilizes diverse trajectories across multiple platforms, which are meticulously labeled via multi-stage human annotation to establish ground-truth verdicts. The researchers also developed OSReward-Hard, focusing on complex edge cases, and OSReward-Multi for fine-grained alignment and efficiency scoring. Their evaluation reveals a critical systemic leniency bias in current VLMs, where even state-of-the-art models frequently misidentify failed tasks as successes. While high-performing judges exist, they are computationally expensive, whereas affordable open models lack sufficient accuracy. To address this, the authors released OS-Shepherd-100K, a large-scale corpus of reasoning-annotated trajectory judgments. Using this dataset, they trained OS-Shepherd (9B and 35B), open-source reward models that provide stable and reliable signals. These models achieve performance comparable to frontier commercial judges while operating at a 30-60x lower cost, offering a scalable solution for CUA reinforcement learning and data curation.