The paper introduces Experience-Handoff Reinforcement Learning (AceRL), a method that co-optimizes an LLM agent's ability to act, reflect, and curate task-local playbooks within a fixed Agentic Context Engineering (ACE) harness. Unlike standard RL from scalar rewards, which discards fine-grained environment data like error logs, and existing reflection-based harnesses where handoff artifacts are hand-designed, AceRL evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts. The approach jointly trains three role-specific behaviors—action generator, insight reflector, and playbook curator—using gradient updates derived from the scalar reward signal. Experiments on agentic benchmarks demonstrate that AceRL achieves higher held-out performance compared to baseline methods. The results indicate that effective agent training requires optimizing not only immediate actions but also the model-generated experience artifacts that subsequent rollouts inherit, establishing a recursive self-improvement loop where the policy learns to produce better context for its future invocations.
Read original
huggingface/daily-papers