Researchers have introduced LLAMIA-Bench, a novel benchmark comprising six diverse collaborative chess tasks designed to evaluate the integration of Large Language Models (LLMs) with non-language agents. These tasks specifically target behavioral imitation, state assessment, and natural-language explanation, presenting problems that neither a standalone LLM nor a specialized chess engine can solve independently. To address the potential "verbalization debt"—the performance loss caused by compressing continuous agent representations into sparse text—the authors propose "latent state internalization." This method projects the subagent's continuous representations directly into the LLM's token stream as learned state tokens, employing dynamic re-encoding as environment states evolve. Experimental results demonstrate that verbalized integration creates a persistent performance gap that increases during training and across model scales from 4B to 14B parameters. In contrast, the researchers' 14B model, LLAMIA, trained via latent state internalization, matches or outperforms task specialists and frontier models like GPT-5.1 when provided with tool access. Furthermore, LLAMIA demonstrates superior out-of-distribution generalization compared to task-specific finetunes, which tended to collapse in such scenarios.
Read original
huggingface/daily-papers