Ubicloud addressed tool-call schema accuracy issues when serving open-weight models like Kimi K2.6 via vLLM. Initial testing with vLLM 0.28 yielded only 92% schema adherence, meaning 2 out of every 25 tool calls failed, leading to degraded agentic performance. To resolve this, they leveraged vLLM’s constrained decoding feature, introduced in v0.25 and stabilized with the `--tool-strict-level` flag in v0.31. Constrained decoding restricts token sampling to only syntactically valid options based on the tool’s JSON schema, effectively turning schema accuracy into a runtime guarantee rather than a probabilistic outcome. This approach uses XGrammar’s structural tags to enforce valid tool call syntax during generation—ensuring proper sequencing through tool call begin, name, index, argument start, schema-constrained JSON, and end. Ubicloud verified 100% tool-call schema accuracy on the K2VV ToolCall benchmark, developed by Moonshot AI, which evaluates 2000 requests across trigger similarity and schema accuracy. Performance overhead was negligible due to GPU-efficient precomputed token masks. The fix elevated their tool-call accuracy from 92% to 100%, aligning open-weight model serving quality with first-party provider standards.
Read original
hackernews