Reddit user u/Significant-Serve-58 trained a 1.17B parameter sparse Mixture-of-Experts model with 348M active parameters on ~11B tokens using a single RTX 5090. The model achieves 85.5% accuracy on shuffled fact chain evaluations, significantly outperforming Qwen2.5-1.5B's 53.3% despite Qwen's 18T token training corpus. The author released GGUF weights and an evaluation set for community validation, emphasizing the model is specialized for factual reasoning rather than general assistance.

Read original