Researchers have introduced Artificial Intelligence System Prompt Assurance (AISPA), a user-centric auditing framework designed to evaluate the instructions developers use to govern Large Language Model (LLM) behaviors. The AISPA framework assesses system prompts across eight distinct dimensions to identify whether instructions are protective of users or problematic to their interests. To validate this method, the authors audited 3,249 instructions extracted from 88 commercial AI products. The empirical analysis revealed significant variability in prompt design; some developers include over 60 protective instructions per product, while others include fewer than five. While 98.9% of products contain at least one protective instruction, only 24% address all eight dimensions of the AISPA taxonomy. Furthermore, the study observed a trend toward longer, more protective system prompts over time. However, problematic instructions remain a persistent issue, with approximately 40% of audited products containing at least one instruction that conflicts with user interests. Notably, protective and problematic instructions frequently coexist within the same prompt, highlighting a critical accountability gap in current AI deployment. The findings suggest an urgent requirement for increased transparency, standardization, and independent oversight regarding the hidden system prompts used in commercial AI applications.

Read original