This study investigates the "lossless" claim of Orthrus, a hybrid autoregressive-diffusion architecture designed to accelerate language model inference through parallel token generation and an intra-model consensus mechanism. While Orthrus aims to produce identical output sequences to a frozen autoregressive backbone, independent reproduction by the researchers reveals that numerical precision significantly impacts its trajectory matching capabilities. Testing across 1,190 prompts from 12 different domains, the researchers found that using BF16 precision resulted in exact trajectory matching in only 45% of cases for the original authors' checkpoint and 43% for an independently trained model. The data indicates that the likelihood of maintaining an exact trajectory is strongly correlated with the response-conditional perplexity of the reference model. Interestingly, while BF16 leads to divergence, switching to FP32 precision achieved exact trajectory matching for all evaluated prompts. Crucially, the study notes that despite these trajectory discrepancies in lower precision, Orthrus does not exhibit systematic performance degradation on downstream lm-eval-harness benchmarks. The findings suggest a critical distinction between mathematical exactness and practical utility, highlighting that the perceived losslessness of speculative decoding methods is highly sensitive to numerical precision and that exact trajectory equivalence should be evaluated independently from downstream task performance.
Read original
huggingface/daily-papers