Researchers have introduced OR-Clarify, a novel benchmark designed to evaluate how Large Language Models (LLMs) handle incomplete natural-language problem descriptions during the optimization modeling process. Unlike previous evaluations that assume complete problem specifications, OR-Clarify addresses the reality of Operations Research (OR) by withholding critical structured slots—such as objectives, constraints, or business rules—and requiring agents to interact with a simulated user to recover them. The benchmark assesses performance across multiple dimensions, including slot recovery accuracy, stopping behavior, the tendency to make silent assumptions, and total interaction cost, supporting both open-ended and choice-based clarification formats. To address these challenges, the authors propose InterOPT, a two-stage Interactive Optimization framework. This method identifies formulation-critical gaps to decide whether the agent should pose a follow-up question or terminate the interaction. Experimental results demonstrate that InterOPT significantly outperforms existing baselines in exact slot recovery within choice-based settings and maintains competitiveness with strong prior methods in open-ended scenarios. By reframing OR assistance as a decision-making process regarding model completeness, this work emphasizes the importance of selective clarification and quantified uncertainty in automated mathematical programming.
Read original
huggingface/daily-papers