The paper introduces "instruction capacity" as a quantitative metric to characterize the degradation of language models' instruction-following ability under increasing load, addressing the lack of a standardized limit expressed in load units with uncertainty. The proposed instrument constructs atomic, independent rules that replace single-token words in text without semantic meaning, where each replacement constitutes a Bernoulli trial with a known position and rule number. A statistical model measures the probability of correct replacement as a function of text length, word position, replacement density, number of rules, and rule number, fitting smooth laws to restore a coding-independent reference level. Evaluated on GPT-4o mini across 2,958 requests and 642,430 trials, the method yielded regular laws for all load factors and answer collapse. The instruction capacity was measured at approximately 37–38 rules for a 0.95 share of correct replacements and 18–19 rules at full replacement density. The instrument code, measurement data, and model passport are openly available, with all passport numbers recomputed from published data points, enabling the passport to serve as a calibration standard for future prompt load measurement.
Read original
huggingface/daily-papers