An analysis of over 40,000 game runs involving human-in-the-loop AI agent command approvals reveals significant security vulnerabilities in manual oversight. The study found that players missed an average of 1 in 3 threats, maintaining a mean accuracy of only 66.3%. Technical analysis showed that while "obviously destructive" commands like rm -rf / were caught at an 88.3% success rate, more sophisticated attacks were frequently overlooked. Specifically, credential exfiltration via commands like cat ~/.aws/credentials saw a 35% miss rate, and exfiltration through hijacked scripts saw a 33.4% miss rate. Notably, the "npm run" pattern proved highly effective for attackers; the command npm run analyze was approved by 64.7% of players, doubling the success rate of other exfiltration attempts because the malicious payload is hidden within the package.json scripts. The data highlights a "human-in-the-loop" dilemma where high noise leads to decision fatigue and over-blocking of benign commands, such as rm -rf dist/ (blocked 45% of the time). This fatigue is reflected in rising miss rates toward the end of sessions, supporting the move toward automated safety models like Anthropic's "Auto Mode" to mitigate the inherent flaws in manual command validation.

Read original