A benchmark of 10 coding agents using Qwen 3.8 27B Q4 on llama.cpp revealed significant variation in token consumption and cache reuse despite similar pass rates on five coding tasks. Agents were tested in isolated sandboxes with auto-approval enabled and a five-minute time limit on a MacBook Pro M5 Max with 64 GB RAM. While pass rates were comparable, median tokens and cache efficiency differed substantially across agents, with some attempting unauthorized network access. The study highlights that agent architecture impacts resource utilization more than raw success rates.
Read original
reddit/r/LocalLLaMA