Researchers have introduced a novel reference-based method for auditing bias in Large Language Models (LLMs) by analyzing internal hidden-state representations rather than relying solely on expensive output-level benchmarks. Because fine-tuning alters the representation geometry, making absolute hidden states incomparable across model versions, the authors propose encoding sentences based on their similarity to a fixed set of anchor sentences. This technique creates relative representations within a shared comparison space, allowing for the measurement of a "Representational Bias Shift" ($\Delta B$), which quantifies how target groups shift in their association with positive or negative attributes. Evaluated across three model families and benchmarks including WildGuardMix, DecodingTrust, and ToxiGen, the $\Delta B$ metric demonstrated strong correlation with output-level bias changes, achieving a correlation coefficient of $|r| = 0.84$ ($p < 0.001$) under full fine-tuning scenarios. The method effectively identified biased checkpoints with ROC AUC values ranging from 0.65 to 0.99, outperforming SEAT-based baselines on the WildGuardMix and DecodingTrust datasets. Notably, this approach is highly efficient, requiring $3\times$ to $50\times$ less computation than traditional output-based benchmarks and completing audits in approximately three minutes. The findings suggest that $\Delta B$ is a stable, task-agnostic metric that serves as a robust complement to existing output-centric auditing frameworks.

Read original