To address the scarcity of high-quality training data for Central Asian languages, a developer created the Multilingual Code and Language Dataset, a 150 GB uncompressed corpus now available on Hugging Face under a CC BY 4.0 license. The dataset is structured into clean .txt and .jsonl files, featuring a heavy emphasis on Russian (60 GB) and significant contributions to low-resource languages, including Kyrgyz (23 GB), Kazakh (22 GB), Uzbek (7 GB), and Tajik (7 GB), alongside 1.5 GB of English. To facilitate easier distribution, the author utilized .7z with a maximum compression profile, a process that required 10 hours of computation and extensive troubleshooting of Out-of-Memory (OOM) errors caused by large dictionary sizes. The final compressed archive was reduced to approximately 27.7 GB. Beyond natural language, the collection includes 20 GB of source code split evenly across Python, C++, Rust, and Go (roughly 5 GB per language). This specialized corpus is designed to support various NLP tasks, such as fine-tuning Code LLMs like StarCoder or CodeLlama to understand technical documentation in local languages, improving technical translation accuracy, and providing sufficient scale for the continual pre-training or scratch training of 1B–2.5B parameter models.
Read original
dev.to