Researchers introduce a novel knowledge-gated task-construction protocol designed to distinguish between an LLM agent's inherent competence and its access to private, domain-specific conventions. By decoupling task instructions from a compact artifact containing private reference tables, utility operators, and specific conventions, the methodology ensures that task instructions remain byte-identical across both provided and withheld-artifact conditions. To ensure rigor, the protocol utilizes construction-time provenance, leak audits, and executable witnesses to verify that task success is strictly dependent on the provided artifact. The study employs fifteen calibration tasks and evaluates performance using deterministic solvers for structured data or named criterion-level rubrics for non-executable outputs. Experimental results demonstrate the effectiveness of this gating; a frontier agent configuration achieved a 68.0% pass rate when the artifact was provided, but dropped to 0% when it was withheld. Furthermore, providing a plausible but incorrect artifact resulted in a 0% success rate across five trials. After applying a configuration-relative calibration screen and a five-trial empirical knowledge-gating screen, the researchers identified seven tasks that successfully satisfy these rigorous validation requirements. While the study validates the construction protocol's ability to isolate knowledge dependence, the authors note that these results do not yet establish improvements in post-training performance.

Read original