RunAnywhere released TrnRT, an open-source inference engine serving GLM-5.3-Flash on a single AWS trn2.48xlarge instance (16 Trainium2 chips) via an OpenAI-compatible API. The engine partitions the 320B MoE model (18B active parameters, FP8) across two tensor-parallel engines, each spanning 32 logical NeuronCores behind a session-affine router. Decode executes as fused NKI megakernels—approximately one per layer half—covering KDA attention, fused DSA layers, mHC, and MoE FFN, designed to keep working sets in SBUF, utilize both physical cores symmetrically, and internalize collectives. A 1M-token context is supported through sparse attention (top 2,048 tokens) and linear attention with fixed-size state. A radix-tree prefix cache manages FP8 KV pages with a host-memory tier for evicted pages and state checkpoints. Offline compilation on any x86_64 host (requiring hours) produces graphs loaded at serve time without on-device compilation. SemiAnalysis AgentX benchmarks at 1M context with 32 concurrent agents show batch 12 (default) achieving 28.9 requests/min, 3.57M tokens/min, 22.3 decode tok/s/user (p50), and 6.53s first-token latency (cached, p50) at $0.040/M tokens on spot ($8.60/h) versus $0.167/M on capacity blocks ($35.76/h). The repository passes graph equivalence across 32 ranks for 157 graphs and byte-identical outputs on 384/384 requests.
Read original
hackernews