Researchers introduce TB-MathModel, the first agentic and verifiable benchmark for full-stack mathematical modeling, built on 18 real-world problems from the 2023–2025 China Undergraduate Mathematical Contest in Modeling (CUMCM) and Mathematical Contest in Modeling (MCM). Each task runs in a containerized terminal providing the verbatim problem statement, official attachments, and figures; the agent must return a single JSON artifact containing the numeric answer to the final question. Scoring uses a direction-aware, one-sided metric against a panel of human Outstanding-award papers—never penalizing answers that beat the reference—and a two-layer feasibility gate: a programmatic checker over six decidable invariant forms and 624 Lean 4 theorem-proving obligations that zeroes infeasible solutions. Evaluated across seven multimodal agents (three seeds each), the strongest system achieves a multi-reference O-Eval score of 64.58%, while a $2.54 open-weight model reaches 59.81% at 1/100th the cost. The benchmark transforms the numeric core of modeling into a reproducible, machine-checkable signal for long-horizon reasoning, enabling systematic collection of agent trajectories. Source code is publicly available.
Read original
huggingface/daily-papers