DLite introduces a history‑free speculative decoding framework designed for agentic AI workloads, where long, evolving contexts and frequent multi‑turn interactions cause conventional draft models to lose up to 80% of their acceptance length while incurring large KV‑cache overhead. By reusing the target model’s contextual hidden states directly, DLite performs local speculative prediction without maintaining an explicit historical context, thereby converting a complex, growing‑sequence modeling task into a simple, block‑wise prediction problem. A lightweight sequential head is added to restore intra‑block causal dependencies with minimal overhead. Training this heterogeneous, history‑free architecture employs a tailored two‑stage strategy that first aligns the draft head with the target model’s representations and then fine‑tunes the sequential head to preserve local dependencies. Evaluated on general‑purpose, long‑context, and agentic benchmarks, DLite achieves an acceptance length of 2 to 6 tokens and outperforms the prior state‑of‑the‑art speculative decoding method DSpark by up to 70% in terms of speed‑up or acceptance‑length improvement, demonstrating substantial inference acceleration without the storage penalty of retaining full history.

Read original