In a recent technical evaluation, the autonomous AI pentesting tool Darkmoon was tested against the OWASP Juice Shop application to assess its efficacy in a black-box environment using entirely local Large Language Models (LLMs). To address privacy concerns regarding data exfiltration to cloud-based services, the experiment was conducted using Ollama and llama.cpp, ensuring that no traffic or code was sent to third-party providers. Running in a Dockerized environment via bkimminich/juice-shop , the autonomous agent successfully identified 57 distinct vulnerabilities within a 28.5-minute window. The breakdown of these findings included 8 critical, 24 high, 21 medium, and 4 low-severity issues, each accompanied by a proof-of-exploitation. The testing demonstrated a compounding effect in discovery over multiple iterations, with the count increasing from 14 to 57 findings across successive runs. This methodology distinguishes itself from white-box benchmarks that often rely on source code access and cloud LLMs, focusing instead on a zero-egress, black-box approach suitable for sensitive production environments. The results and methodology have been documented in a community benchmark repository to facilitate reproducible testing for other autonomous security tools.
Read original
dev.to