Sarvam has launched Sarvam Token Factory, a managed inference platform serving eight open-weight models—including Sarvam 105B (105B total, 9B active, 128K context), GLM 5.3, GLM 5.3 Flash, DeepSeek V4.1 Flash 1017, DeepSeek V4 Pro, Kimi K3, Kimi K3.1 Pro, and Gemma 4 31B—from NVIDIA B200 GPU clusters in India via a single OpenAI-compatible API, with on-premise deployment available for enterprises. The stack comprises a cache-aware control plane (router, admission control, planner), separate prefill and decode GPU pools, and a shared cache layer combining GPU memory with a G3 store built on Mooncake, leveraging vLLM, SGLang, Dynamo, Mooncake, NIXL, DeepEP, DeepGEMM, and FlashInfer. Caching drives performance: DeepSeek V4.1 Flash serves 450B+ tokens daily with 95.7% of prompt tokens cached; on GLM 5.3, the planner reduced prefill GPU-hours from 32.3 to 17.1 under replayed load; and Sarvam 105B uses MXFP8 weights plus speculative decoding to raise output throughput ~2.2x, fitting one replica per B200. Self-reported benchmarks show Sarvam 105B at 0.2s first token and 330 tok/s (single user); Gemma 4 31B at 0.40s/308 tok/s (10K prompt, Mumbai) versus 2.00s/193 tok/s elsewhere; Kimi K3.1 Pro at 6,177 tok/s/GPU versus 3,213 for InferenceX at 70 tok/s/user (1.9x); 72-session agent load median first token 0.8s versus 11.6s; GLM 5.3 Flash p99 first token 3.2s versus 14.1s for stock vLLM/Dynamo. Together AI leads on DeepSeek V4.1 Flash 10K prompt (0.71s vs 0.90s first token; 278 vs 130 tok/s Standard). Introductory pricing (through 2026): Sarvam 105B ₹10/₹25 per 1M input/output; GLM 5.3 ₹126/₹390; GLM 5.3 Flash ₹13.5/₹45; Gemma 4 31B ₹35/₹90; DeepSeek V4.1 Flash ₹14/₹55. Sarvam-measured TTFT from Mumbai: 1K prompt 0.51s vs Together 0.61s, Fireworks 1.01s, DeepInfra 1.08s; 10K prompt 0.90s vs Together 0.71s, Fireworks 1.55s, DeepInfra 1.75s. All figures are self-reported.
Read original
marktechpost