Researchers have introduced ActObs, a novel supervised fine-tuning (SFT) approach designed to improve reinforcement learning (RL) performance by supervising both action and observation tokens within agent trajectories. While standard SFT typically applies loss only to action tokens, ActObs forces the model to predict the subsequent environment observations, encouraging the policy to internalize the consequences of its actions without increasing parameter count, data, or computational overhead. Experimental results on the Terminal-Bench 2.0 benchmark demonstrate that when using Group Relative Policy Optimization (GRPO), ActObs-initialized models outperform action-only models across various sampling budgets. Specifically, on Qwen3-4B, ActObs achieved a higher pass@k at every evaluated budget, while on Qwen3-8B, it showed a +3.4 percentage point increase in pass@16. The method also proved effective for cross-domain tasks, such as aider-polyglot, where it improved pass@1 by 4.2 percentage points on unseen code editing tasks. The study concludes that action-only SFT causes observation prediction capabilities to degrade below base model levels because action and observation gradients become orthogonal. In contrast, ActObs maintains higher entropy and requires less policy movement during RL, preserving the model's ability to model environmental consequences and facilitating more effective downstream exploration.

Read original