Surge AI released DAYJOB: Tax, a benchmark evaluating AI agents on 50 professional tax assignments across eight practice areas including individual, partnership, corporate, international, and estate tax. The benchmark comprises 1,789 workspace files (median 30 per task) — PDFs, Excel workbooks, IRS notices, ledgers, contracts, and unstructured evidence like handwritten receipts — created and validated through a three-layer review by tax experts from PwC, Deloitte, Baker Tilly, and EY. Unlike TaxBench, TaxCalcBench, or TaxLab, which test defined questions, calculations, or information gathering, DAYJOB measures end-to-end professional judgment: identifying consequential issues the client never raised, resolving conflicting evidence, and delivering complete workpapers, returns, or advisory letters. Tasks require agents to recognize when records are insufficient, such as a missing receipt invalidating a reimbursement, or a dispatch log revealing multistate nexus contrary to a client's assumption. In a highlighted case, an agent correctly determined that selling a condo in October 2026 or January 2027 satisfied the two-of-five-year residence exclusion for $250,000 in gains, but failed to flag that eligibility would expire around August 2027 based on the client's August 2024 move-out. Across five attempts, Opus 5.5 identified this deadline every time; Astra never did. On the leaderboard at pass@1 with a 95% rubric threshold, Claude Sonnet 5.5 leads at 10.8%, followed by Fable 5.1 (9.6%) and Opus 5.5 (9.2%). Sonnet 5.5 and Opus 5.5 define the high-performance Pareto frontier against average cost per trial. Results indicate frontier models still struggle to reliably exercise professional judgment across realistic tax workflows.
Read original
hackernews