This study addresses the limitations of small-scale language models in interactive dialogue games, specifically focusing on their inability to maintain state, interpret feedback, and execute valid actions. Using a 2B open-weight model in the LM Playschool Challenge, the researchers identified that failures often stem from local decision errors—such as malformed actions or ignoring recent feedback—rather than just general knowledge gaps. To mitigate this, the authors propose a three-stage post-training recipe: acquiring broad participation through supervised fine-tuning (SFT), repairing specific mechanical failures via turn-local preference pairs within a target dialogue-game family, and preserving general capabilities to avoid catastrophic forgetting. The results demonstrate significant performance gains in targeted domains, with the public clemscore rising from 10.67 to 38.92 and the closed in-domain score increasing from 13.41 to 41.17. Notably, the model maintained its aggregate static performance, shifting only slightly from 44.14 to 44.24. However, the benefits of this targeted repair were largely confined to the specific game family, as the out-of-domain clemscore remained low at 7.88. The findings suggest that while SFT provides the foundational capability, precise turn-local supervision is an effective method for correcting specific behavioral errors, provided the transfer is expected to remain within the targeted domain.

Read original