Momentic engineering lead Jeff An argues that coding agents fail at self-verification due to three systemic flaws: shallow checks, shared blind spots, and context-driven completion pressure. An audit of 1,000 AI-authored unit tests revealed only 9.6% were useful verification checks; 33% tested low-consequence logic, 22.8% duplicated coverage, and 22.4% over-relied on mocks. Kun Chen's experiment across 111 tasks showed agents writing their own tests achieved a 65.3% success rate versus 66.2% for those prohibited, indicating self-authored tests provide negligible signal. Shared blind spots cause agents to test implemented behavior rather than requirements; a 2026 study confirmed self-written tests often accept incorrect patches. Context rot exacerbates reliability: Chroma's 2025 study found Claude Sonnet 4 accuracy dropped from 89% to 38% as context grew from 300 to 113,000 tokens, while Anthropic observed a "desperation" vector driving models to declare completion prematurely after repeated failures. Momentic's solution decouples generation from verification using an independent QA agent ("Mo") that exercises live user workflows via remote browsers and mobile devices, starts with fresh context and original requirements, and leverages five model providers to mitigate self-preference bias. Benchmarked on four open-source apps, Mo detected 3-5x more bugs than Codex or Claude Code operating autonomously. The article posits "verification engineering" as a distinct discipline where ROI shifts from unit test volume to end-to-end defect detection rate per unit time and cost.
Read original
stackoverflow/blog