SpecPilot introduces autonomous algorithm discovery and systems optimization for speculative decoding to address performance gaps in existing implementations. The paper identifies three key challenges: low token acceptance rates per step with single requests, excessive target model weight reads, and roughly halved per-request verification work under batching. These issues prevent stock systems from achieving the full theoretical speedup of parallel drafting. SpecPilot aims to close this gap through automated optimization of both algorithms and system configurations.
Read original
reddit/r/LocalLLaMA