Memorilla introduces a latent semantic memory system that enables frozen LLMs to access external knowledge by compressing document collections into a fixed set of query-dependent memory tokens, avoiding full-context inference. The system encodes documents—such as user conversations or book chapters—once into reusable embeddings, then retrieves and projects query-specific information into the model's input embedding space using only 16 memory tokens. Evaluated across eight benchmarks covering personalization, long-document QA, and knowledge-intensive tasks, Memorilla outperforms baselines including Mem0 and MemoRAG on five. On FactKG, it achieves 84.33% accuracy versus 80.43% for full-context inference while reducing token representation of source documents by 108x. In a TextWorld agentic environment, Memorilla supports continual learning as the document count grows with the agent's trajectory, reducing redundant room revisits and raising the average success rate from 18.6% with full-history in-context learning to 24.2%. The approach demonstrates effective knowledge accumulation and utilization for frozen LLMs across extended interactions, advancing continually learning AI systems.