Buoyant released Kinference v0.1.0-alpha.7, an Apache 2.0-licensed, Rust-based inference control plane for Kubernetes and vLLM that manages routing, token rate limits, KV-cache-aware load balancing, and per-tenant metrics via five CRDs: GatewayBackend, InferencePolicy, InferencePool, InferenceProvider, and InferenceWorkerSet. The system comprises a traffic gateway and an inference handler exposing a gRPC interface, which consumes vLLM KV-cache events and engine metrics using NVIDIA's dynamo-kv-router crate to predict time-to-first-token (TTFT) per endpoint. Declarative InferencePolicy rules match on tenant, model, GPU pool state, and semantic tags to implement patterns such as routing simple prompts to smaller models unless cache overlap favors larger ones, bursting to external providers like Bedrock when local queues exceed two requests, enforcing PII data locality with fail-closed rejection, admitting batch jobs only to workers under 70% KV-cache usage, and aliasing stable model names across multiple backends. Benchmarks on three L40S GPUs running vLLM 0.31.0 with Qwen3.8-27B-FP8 and GuideLLM 0.8.1 load showed median TTFT dropping 95% from 6.88 s to 0.31 s on shared-prefix workloads and 65% from 1.70 s to 0.59 s on multiturn traffic versus a ClusterIP Service baseline using random kube-proxy routing; control traffic without shared prefixes saw no median TTFT improvement (6.92 s vs 6.83 s). Peak goodput under TTFT SLOs reached 103.0 vs 14.0 req/min (7.4x) on shared-prefix and 82.7 vs 23.8 req/min (3.5x) on multiturn. Kinference differentiates through governance features—policy CRDs, provider fallback, model aliasing, and tenant accounting—while llm-d and NVIDIA Dynamo lead on disaggregated prefill/decode and broader engine support. Installation requires Gateway API v1.5.1 CRDs, Buoyant Gateway Helm chart, and Kinference chart, shipping Prometheus metrics and Grafana dashboards for TTFT, ITL, and cached token counts per tenant.

Read original