JuliaHub conducted a comparative evaluation of frontier AI models—specifically OpenAI’s GPT-5.6 family (Terra, Sol, and Luna) and Anthropic’s Claude-Fable-5—to determine their efficacy in physical AI modeling and simulation. Using the Dyad AI agent harness, the study tested 52 graded runs across five complex engineering problems, including a high-difficulty NASA HL-20 flight vehicle simulation. Unlike standard benchmarks, grading was based on simulated trajectories compared against sealed ground truth rather than code compilation success. Claude-Fable-5 emerged as the performance leader with a difficulty-weighted score of 0.889, achieving a perfect 12/12 on core problems and leading on the HL-20 task by utilizing extensive falsification and external reference checks. However, Fable was significantly more expensive at $9.60 per trial. GPT-5.6 Sol provided the highest value, securing a 0.814 score at only $1.74 per trial. GPT-5.6 Terra was the most economical ($1.25/trial) but occasionally sacrificed precision, while Luna performed worst across all metrics. Crucially, the study found that the specialized Dyad harness provided more significant performance gains than model selection alone; a single model's score jumped from 0.533 in a stock coding agent to 0.899 within the Dyad harness, demonstrating that specialized workflow enforcement is more critical for physical accuracy than model scale.

Read original