A developer successfully transferred Qwen3.8-Flash-Next's pretrained PLE n-gram memory to a smaller Qwen3.5-0.8B model, achieving a 5.05% reduction in validation perplexity without fine-tuning the backbone. The method uses frozen 51B-parameter PLE memory and a lightweight R=1 reader with a token-dependent linear gate at decoder layers 3 and 9, trained using limited Kaggle GPU resources. This approach demonstrates effective knowledge distillation for compact language models.

Read original