Researchers introduce PIPETTE, a unified prompt injection purple-teaming framework addressing the gap between high attack success rates (ASR) in red-teaming and actual robustness gains from adversarial training for LLM agents. The authors identify that state-of-the-art RL-based red-teaming methods RL-Hammer and PISmith suffer from mode collapse, repeatedly generating similar attack strategies that yield diminishing robustness returns. PIPETTE resolves this through a dictionary learning-based reward mechanism that incentivizes broader coverage of distinct attack strategies during red-teaming, combined with an on-policy adversarial training pipeline specifically tailored for indirect prompt injection (IPI) defense. Evaluated on agentic benchmarks AgentDojo and ASB, defenders trained with PIPETTE achieve near-zero ASR against static attacks and withstand strong adaptive attacks while preserving benign task utility comparable to undefended baselines. The framework demonstrates that effective purple-teaming requires not just attack diversity but also tight integration between offensive discovery and defensive adaptation to convert red-teaming findings into measurable robustness improvements for deployed LLM agents.
Read original
huggingface/daily-papers