The article argues that teams relying directly on hosted LLM interfaces surrender the observability needed to understand inference behavior, recommending a self-hosted proxy as an instrumentation and control layer. A provider’s token-usage display is characterized as a generic proof-of-work indicator rather than an engineer-oriented time series, while OpenTelemetry export alone is considered inadequate. The proposed proxy should expose one controlled endpoint for multiple providers and models; record the API key, selected model, client, timestamp, and request status; and issue virtual API keys so workloads, agents, and harnesses do not share a primary credential. Because the proxy is operator-controlled, model or provider changes can be made quickly without rebuilding the surrounding stack. It can also be customized with richer analytics, custom MCP server endpoints, and other features. The author reports tracing the cause of a coding-session cache miss manually in 20 seconds and notes that the proxy API could automate this investigation. More broadly, operating the proxy provides control over infrastructure, application-level visibility, request routing, and storage. Example use cases include automating image-to-CAD work with ComfyUI and querying a model for food macronutrient information, illustrating that a proxy can govern varied AI-assisted applications rather than merely provide chat access.

Read original