SpecFirst introduces a novel two-stage framework designed to improve agent-based program synthesis from scratch, specifically addressing the limitations of single-pass LLM approaches that struggle with ambiguous documentation and insufficient behavioral probing. Recognizing that existing models solve less than 1% of instances on the ProgramBench benchmark when provided only with natural-language documentation and an execute-only binary, the authors propose decoupling the process into two distinct phases: behavioral specification elicitation and code synthesis. In the first stage, a dedicated "spec agent" probes the binary oracle and integrates these observations with the provided documentation to generate a structured behavioral specification. In the second stage, a "code synthesis agent" utilizes this formalized specification to implement the program. This architecture prevents context drift and ensures that ambiguities are resolved before implementation begins. Experimental evaluation conducted on all 200 ProgramBench instances across four different LLMs demonstrates that SpecFirst significantly outperforms single-loop baselines. The framework achieved statistically significant improvements, increasing test pass rates by 6.9% to 21.3% and enhancing binary exploration coverage by 9.4% to 18.5%. Furthermore, behavioral analysis indicates that this explicit requirements-engineering phase facilitates earlier and more sustained code construction, proving that structured specification elicitation is a highly effective paradigm for complex, from-scratch software engineering tasks.
Read original
huggingface/daily-papers