Microsoft's VibeVoice is an open-source research framework featuring advanced speech AI models, primarily focusing on long-form Text-to-Speech (TTS) and Automatic Speech Recognition (ASR). A core technical innovation is the use of continuous acoustic and semantic speech tokenizers operating at an ultra-low frame rate of 7.5 Hz, which enhances computational efficiency for long sequences. The TTS component utilizes a next-token diffusion framework, combining a Large Language Model (LLM) for contextual understanding with a diffusion head for high-fidelity acoustic generation. This allows for synthesizing up to 90 minutes of multi-speaker audio (up to 4 speakers) in a single pass. The VibeVoice-ASR model supports single-pass processing of up to 60-minute audio files, providing structured transcriptions that include speaker identification, timestamps, and content, alongside support for customized hotwords and over 50 languages. For edge deployment, the VibeVoice-ASR-BitNet engine employs heterogeneous quantization (I8_S + I2_S) to compress the model from 4.62 GB to 1.58 GB, enabling real-time inference (RTF < 1) on 3+ CPU threads without requiring a GPU. Additionally, the VibeVoice-Realtime-0.5B model provides low-latency TTS with approximately 300 milliseconds of first audible latency.
Read original
github-trending/python