TAO introduces a tool-aware joint scheduling framework for LLM-based agentic serving that addresses inefficiencies in existing systems, which ignore tool-return readiness and rely on heuristic GPU admission and placement. The framework's value interface integrates three components: near-term tool readiness, node-specific cache credit with time-aware decay, and token footprint. A dynamic program jointly selects and places waiting sessions under GPU capacity constraints. Evaluated across 12 model-workload-concurrency settings, TAO reduces latency by up to 26.8%, cuts mean time to first token (TTFT) by up to 36.3%, and increases decode throughput by up to 29.5% compared to the best state-of-the-art baseline for each metric.

Read original