VoxCPM2 is a 2‑billion‑parameter tokenizer‑free text‑to‑speech system that generates speech directly in the latent space of AudioVAE V2 through a diffusion‑autoregressive pipeline (LocEnc → TSLM → RALM → LocDiT). Trained on more than two million hours of multilingual audio, it supports 30 languages without language tags and produces native 48 kHz studio‑quality output. The model enables voice design from a natural‑language description (e.g., “a young woman, gentle and sweet voice”), controllable cloning from a short reference clip with optional style guidance, and ultimate cloning when both reference audio and its transcript are supplied, preserving timbre, rhythm, emotion and style. On an NVIDIA RTX 4090 the standard PyTorch implementation achieves a real‑time factor (RTF) of ~0.30, which drops to ~0.13 when accelerated with Nano‑vLLM or vLLM‑Omni, requiring roughly 8 GB of VRAM. Benchmark results show Seed‑TTS‑eval WER of 1.84 % (English) and 0.97 % (Chinese) with SIM scores of 75.3 % and 79.5 %; hard‑set EN WER is 8.13 % (SIM 75.3 %). Internal 30‑language ASR evaluation yields an average error rate of 1.68 %. Instruction‑guided voice design scores APS 85.2/DSD 71.5/RP 60.8 for Chinese and APS 84.2/DSD 83.2/RP 71.4 for English. The code and weights are released under Apache‑2.0, with serving options via Nano‑vLLM‑VoxCPM, vLLM‑Omni, and llama.cpp‑omni for CPU/Metal/CUDA/Vulkan deployment.
Read original
github-trending/python