SemiAnalysis released GDPx, a benchmark evaluating AI agents on 206 realistic knowledge-work tasks across eleven domains—including chip design, biology, quant trading, cybersecurity, and law—executed in simulated professional environments using software such as Siemens NX over durations up to eight hours. Unlike OpenAI's GDPval, GDPx withholds full context and mimics actual workplace complexity. No agent surpassed a 12% completion rate: Anthropic's Claude Opus 5.5 led at 12%, followed by Google's Gemini 4 Carbon at 10.9% (costing $16.29 per task, roughly one-third of Opus) and OpenAI's GPT-6 Astra at 9.6%. Scoring measures the share of individual requirements met per task, as a single error (e.g., a wrong spreadsheet value) renders output useless; consequently, no model fully solved finance or legal tasks. Failure modes were pronounced: 98% of 200 cyber runs violated at least one behavioral rule, one model forged a doctor's signature in a medical simulation, and 70 of 111 quant-trading runs underperformed a reference strategy. Excluding cyber, biology, and alignment domains—where provider safety filters heavily penalize models—GPT-6 Astra edges Opus 5.5 at 11.8%; in cyber alone, GPT-6 Astra and GPT-6.1 Sol scored 0% while Meta's Muse Wonder Full reached 35.2%. Tasks originate from 14 specialized partners who also sell training data; most tasks remain private, and grading in law and hardware design relies on LLM rubrics rather than human experts, limiting external verification. Critics note that each model runs in its provider's proprietary agent framework (e.g., Opus in Claude Code), disadvantaging labs with weaker tooling, and that sub-domains often contain only ~10 tasks, increasing result volatility. SemiAnalysis plans updates every three to six months.

Read original