Grid: An Open-Source Unified Proxy for Distributed Local LLM Inference

Grid is a lightweight, open-source routing layer designed to unify multiple local inference engines—including Ollama, vLLM, LM Studio, MLX, and ComfyUI—across a Local Area Network (LAN) into a single, manageable endpoint.

Simplifying Distributed Local Inference

Managing local Large Language Model (LLM) deployments across multiple machines often introduces significant operational overhead. Developers frequently face the tedious task of manually SSH-ing into various nodes to synchronize CUDA versions or manually sharding models across available hardware. Grid aims to eliminate this friction by providing a unified interface that abstracts the underlying hardware distribution.

Technical Architecture and Implementation

Grid is engineered for minimalism and high performance, focusing on routing rather than orchestration. The core implementation is characterized by the following technical specifications:

• Lightweight Footprint: The project consists of approximately 3,000 lines of Python code.
• Asynchronous I/O: Built using asyncio and httpx to ensure non-blocking request handling and low-latency proxying.
• Stateless Design: The system avoids the complexity of a persistent database. Instead, the system state is maintained dynamically via heartbeats from the connected engines.
• Non-Intrusive Routing: Grid operates as a proxy; it does not restart or modify the underlying inference engines. It simply routes requests to the appropriate backend.

Fault Tolerance and Performance

To ensure reliability in a LAN environment, Grid implements a health-monitoring system. If a specific inference engine goes offline, Grid automatically marks the node as unavailable and reroutes traffic to active nodes, preventing request failures. The overhead introduced by the proxy layer is minimal, with latency added in the tens of milliseconds range.

Supported Engines

Grid provides a single endpoint that can route requests to various popular backends, including:

• Ollama and vLLM (High-throughput serving)
• LM Studio (Local GUI-based inference)
• MLX (Apple Silicon optimized)
• ComfyUI (Node-based generative AI workflows)

Note: As this information is based on an initial community announcement, detailed documentation regarding the load-balancing algorithms and specific API compatibility layers is currently limited.