NVIDIA-NeMo has released Switchyard, an open-source library designed to optimize AI agent workloads by routing requests between efficient and high-capability models. By balancing task success, cost, and latency, Switchyard allows developers to manage a diverse model pool through various integration methods, including gateway plugins for LiteLLM (version 1.102.0) and NeMo Relay (versions 0.8.0 to 0.9.0), a standalone OpenAI/Anthropic-compatible proxy, or direct embedding into Rust or Python applications. The framework offers four primary routing strategies: "Auto" (a preset using efficient-first execution with a 0.5 confidence threshold), "Task" (utilizing an LLM classifier to judge model capability), "Execution" (using recent tool activity and outcome signals via a stage router), and "Composite" (combining task classification with execution signals). While the "Auto" preset is fixed in v0.3.0, the library supports more granular control via TOML configurations. Currently, the ecosystem includes several components at varying stability levels: the beta-stage switchyard-libsy for embedded routing, the alpha-stage switchyard-llm-client for protocol translation, and the alpha-stage switchyard-runner for executing routes within runtimes like NeMo Relay. Users are advised to pin versions due to the pre-1.0 development status of the APIs.

Read original