Researchers have introduced NOLLI, a procedurally generated puzzle benchmark designed to diagnose the performance disparities between English and Korean in large language models. The benchmark consists of 7,500 items across 25 tasks and 15 puzzle types, utilizing behavioral difficulty calibration to ensure items fall into specific accuracy bands rather than relying on structural complexity. The evaluation framework employs a three-level hierarchy: matched direct translations, script adaptations involving Hangul jamo, and Korean-only tasks focused on culture or orthography. Testing 15 frontier and open-weight models revealed that while overall English-Korean accuracy is statistically comparable within a +/- 10 percentage point margin, significant gaps emerge in writing-system-intensive tasks. Specifically, Korean Cipher accuracy lags behind English by as much as 68.7 percentage points, whereas Cryptarithmetic performed over jamo showed no systematic penalty. The study found that Jamo Composition accuracy serves as a predictor for Korean Cipher performance, suggesting that deficits in multi-step sub-syllabic execution drive the language gap. Furthermore, while most models showed varied performance on Korean-only rule-application tasks, all 12 models exceeding a 3% accuracy floor exhibited a consistent deficit in kinship-related tasks. Notably, the researchers determined that structural item size is an unreliable proxy for empirical difficulty, as it failed to scale from easy to hard in 7 of the 15 puzzle types.
Read original
huggingface/daily-papers