A developer built a fully offline modular voice agent that runs on an RTX 5080 laptop GPU, employing a 3‑bit quantized Qwen3.6‑35B‑A3B MoE model (~3 B active parameters) via llama.cpp, Whisper for speech‑to‑text, Piper for TTS, and a local graph memory for context. End‑to‑end latency from end‑of‑utterance to first audio ranges from 1.4 to 3.5 seconds, with a cold‑load time of about 25 seconds and token generation around 98 tok/s. No API calls, cloud services, or network traffic are used at runtime.
Read original
reddit/r/LocalLLM