Vercel Labs has open-sourced agent-desktop 1.0.2 under Apache-2.0, a Rust CLI and C library that enables AI agents to control native desktop applications on macOS 13+, Windows 10 1809+/Server 2019+, and Ubuntu 24.04+ (x64 and ARM64) without vision models. Instead of screenshots, it reads OS accessibility trees — AX API on macOS, UI Automation on Windows, AT-SPI on Linux — returning JSON snapshots with qualified refs (e.g., @s8f3k2p9:e8) for every actionable element. The tool exposes 60 commands (56 operational) via a fixed chain: the CLI or C library calls agent-desktop-core, which uses a PlatformAdapter trait to reach the native API; each binary targets exactly one OS. Ref resolution is strict and fails closed; actions return STALE_REF or AMBIGUOUS_TARGET when UI changes, requiring a fresh snapshot. Skeleton snapshots traverse only three levels deep, reporting children_count and drill refs for containers; snapshot --root then reads one region. Self-reported token reductions range from 78% to 96% on dense Electron apps (Slack, VS Code, Notion), with Apple Maps dropping from 3,811 to 273 tokens per look and a simulated 52-step task falling from 198k to 31k tokens. Actions are headless by default — no pointer movement or focus stealing — with a --headed flag for physical input; ref actions auto-wait up to 5,000 ms for visible, enabled, stable targets. Installation via npm bundles native binaries; macOS requires an Accessibility grant, Windows needs no grant at the terminal's integrity level, Linux requires a graphical session with at-spi2-core. The C library supports in-process calls from Python, Swift, Go, Ruby, and Node. Unlike pixel-based approaches (Anthropic: ~1,000–1,800 tokens per screenshot plus ~4,500 toolset tokens; OpenAI: undisclosed), agent-desktop is model-agnostic, returns structure, and runs on plain CPU with no GPU requirement. Apps lacking an accessibility tree remain unsupported.
Read original
marktechpost