Open sourcing InfiniteKV: a KV cache that files old tokens as 104-byte searchable records in RAM or on disk instead of deleting them. Mistral-7B answered from token 76,747, 2.3x past its trained window. Colab demo

u//u/Final-Data-1410 2026-06-12

Open Sourcing InfiniteKV: Extending LLM Context via Searchable KV Cache Offloading InfiniteKV introduces a novel approach to KV cache management by storing aged tokens as compact, 104-byte searchable records in RAM or on…

→ View original source
Loading more articles...