APort Vault is a benchmark that evaluates payment authorization in tool-using AI agents by replaying 4,371 human-written attacks against a live payment agent during a public capture-the-flag event. It tests 14 models from eight laboratories across five policy configurations and two replay tracks, both with and without a deterministic pre-action check implementing the Open Agent Passport specification, completing 225,964 evaluations.

Read original