Researchers demonstrate that reinforcement learning (RL) post-training gains in diffusion transformers are largely attributable to a constant activation shift rather than complex input-dependent transformations. Testing on Stable Diffusion 3.5 Medium (SD3.5-M) and video models, they fit fixed steering vectors added to frozen base model block outputs to match the velocity predictions of six public RL-tuned image checkpoints and two video checkpoints, using no reward signal during fitting. These vectors contain only 0.29M parameters—65× fewer than the 18.8M LoRA parameters in the RL-tuned models—yet recover 90.0–96.3% of reward gains on image models and 89.8% and 111.9% on video models across metrics including OCR accuracy for text rendering and PickScore for human preference. When initialized at zero and trained directly with online RL, the steering vectors alone exceed the reward improvements achieved by weight-based RL on both OCR and PickScore benchmarks. The results indicate that policy-gradient RL post-training primarily steers the pretrained generator toward capabilities it already possesses via a prompt-independent bias, obviating the need for learned input-dependent transformations for substantial quality gains.
Read original
huggingface/daily-papers