Stanford's SNAP group, with collaborators from Toronto, Penn, and MIT, has open-sourced Memorilla, a 194M-parameter latent memory module that compresses arbitrary document collections into 16 fixed soft tokens for a frozen Qwen3-8B decoder. A frozen Qwen3-Embedding-4B encodes documents once; per query, two 8-head cross-attention layers attend 16 learned slots to the question and the full embedding store, projecting the result into the decoder's input space. Training spans three stages—1.98M Wikipedia paraphrase-reconstruction examples, 1.34M QA/summarization samples, and benchmark fine-tuning—completed in roughly 19 hours on four H100 GPUs. On eight benchmarks using the same frozen Qwen3-8B, Memorilla achieves the highest reported average score of 49.68, leading on five: FactKG accuracy reaches 84.33% versus 80.43% full-context and 80.38% MemoRAG with 108× fewer document tokens; PersonaMem-v2 hits 44.60% versus 30.90% xRAG; TriviaQA exact match 57.00% versus 56.60% MemoRAG; LaMP-4/7 ROUGE-L 18.15/48.37. It trails on PubMedQA (59.00% vs. 69.00% LLMLingua-2), PersonalizationV4 (66.65% vs. 67.88% MemoRAG), and NarrativeQA ROUGE-L (19.31 vs. 20.51 MemoRAG), where MemoRAG uses 50–690× more tokens. Compression peaks at 4,212× on NarrativeQA (16 tokens replace 67,385). Inference on one H100 takes 0.74 s at 512 documents (128K tokens) versus 19.9 s full-context; an optional KV cache yields 5.0 ms/query at 1M documents on a B200 with torch.compile (80 GiB at 4M). Code, checkpoints, and the PersonalizationV4 dataset (149 users, 200 conversations each) are released under Apache 2.0 and CC BY 4.0.

Read original