Researchers introduced LittleLearner, a series of language models (0.6B, 1.3B, and 5B parameters) designed to study knowledge acquisition by constraining their training to a specific pedagogical boundary. The models were trained from scratch on LittleCurriculum, an 88B-token corpus distilled from FineWeb-Edu using a five-stage pipeline aligned with U.S. Common Core K–5 standards. This dataset explicitly excludes any concepts, facts, or vocabulary taught above the fifth grade. To ensure rigorous comparison, each LittleLearner model is matched with an unfiltered control sharing the same architecture and training recipe. Experimental results indicate that the pretraining data sets a definitive capability ceiling; while scaling model size, applying Supervised Fine-Tuning (SFT), using Group Relative Policy Optimization (GRPO) post-training, and employing in-context learning all amplify in-scope K–5 abilities, none of these interventions meaningfully improve performance on out-of-scope tasks. Specifically, GRPO post-training on MathCAMPS boosts K–5 mathematical proficiency but fails to recover advanced reasoning beyond the fifth-grade threshold. These findings suggest that standard interventions primarily elicit or amplify existing knowledge within the training distribution rather than enabling the acquisition of entirely new, out-of-scope reasoning capabilities.

Read original