Researchers have introduced IdeaAMBIG, a new benchmark designed to evaluate the "codification readiness" of research-method specifications—specifically, whether a written method provides enough detail for a human or AI agent to implement it without making unsupported assumptions. The benchmark consists of 660 evidence-grounded instances, comprising 163 real-world gaps extracted from GitHub issues and reproducibility reports, alongside 497 controlled synthetic gaps injected into otherwise complete references. IdeaAMBIG assesses three distinct capabilities: codification-readiness assessment, defect localization (identifying where a specification is insufficient), and clarification action generation (proposing ways to resolve the ambiguity). Testing across 13 Large Language Models (LLMs) revealed that defect localization is the primary technical bottleneck. The top-performing model achieved a Macro Defect Recovery Rate of only 9.6% on real-world instances, though its Macro Clarification Action Success Rate reached 80.6% when provided with the annotated defect. An oracle study demonstrated the severity of these implementation gaps: providing a model with the gold resolution increased the downstream codification-ready rate from a baseline of 14% to 98%, highlighting the significant disconnect between high-level research ideas and the technical specificity required for faithful reproduction.
Read original
huggingface/daily-papers