The "Ready Cohorts" framework optimizes LLM-agent execution by offloading control-path routing decisions directly to the GPU. By utilizing four-byte, GPU-computed routing, the system avoids costly round-trips to host memory for redispatching between LLM and external tool calls. Benchmarking across four GPU models demonstrated that this on-device approach outperforms host-dispatching in all 36 tested settings, achieving a row-median speedup between 1.19x and 2.39x while maintaining 100% accuracy across over 14.5 million calls. To maximize GPU utilization, the authors introduce a cohort-based mechanism that batches agent transitions within specific launch time windows. The research evaluates GPU-active time shares through four metrics: a fixed-partition baseline (F), an optimal partition (P*) derived via dynamic programming, an upper bound (U), and an online result (A). Simulations using a Poisson-stationary 851-session trace—modeled with 100,000 target active agents and a 50 ms launch deadline—showed that while the baseline F achieved 30.19% utilization, the optimal P* reached 43.00% compared to an upper bound U of 45.85%. Notably, the researchers found that 81.83% of the potential optimization opportunity can be recovered through cohort packing at fixed window ends, proving that deterministic agent control paths can be effectively accelerated through temporal and agent-based batching.

Read original