The APort Vault benchmark evaluates payment authorization security in tool-using AI agents by replaying 4,371 human-authored attacks against a live payment agent. The study utilized 14 different models from eight labs, conducting 225,964 total evaluations across five policy configurations and two replay tracks. A central component of the research is testing the Open Agent Passport (OAP) specification, a deterministic pre-action check designed to secure the authorization boundary. The attack distribution varied by difficulty level, with 79.4% of evaluations occurring at Level 4. While model-alone evaluations allowed 140 unauthorized transfers to unpermitted recipients out of 76,842 attempts, the implementation of the OAP layer reduced this figure to zero across 69,297 evaluations. Specifically, within matched model, prompt, and track triples, the layer successfully prevented 105 unauthorized transfers. This security was achieved not through broad payment refusals, but through targeted policy enforcement; of 25,640 transfer calls evaluated by the policy, it denied 187, including 148 calls involving forbidden recipients. The data shows that the OAP layer maintained a per-session upper bound of unauthorized transfers at 0.38%, demonstrating its efficacy in enforcing recipient allowlists without disrupting legitimate transactions.
Read original
huggingface/daily-papers