LiveFoley introduces a streaming causal autoregressive diffusion architecture for real-time video-to-audio generation that closes the gap with offline models by addressing two core bottlenecks. For the visual frontend, it replaces fixed look-ahead with a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining; realigning the window preserves precise audio-visual alignment while a configurable hop size reduces frontend latency below the human perceptual threshold. This tunable hop size underpins the Unified Varhop Framework, which adapts to diverse latency and compute budgets. For the audio backend, LiveFoley widens the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT injects past context to improve quality at zero latency cost and serves as the default configuration. Future-Warmup LocDiT breaks the autoregressive deadlock by supplying an approximate future condition, further boosting generation quality at the cost of an acceptable backend look-ahead. Extensive experiments show LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality, achieving imperceptible end-to-end latency.
Read original
huggingface/daily-papers