Zero-Mem introduces a novel framework for Large Language Model (LLM) agents that utilizes "zero-token memory operations" to manage long-term interactions without the overhead of intermediate LLM calls. Unlike traditional systems that consume tokens to generate and mediate memory records, Zero-Mem preserves original interaction traces directly. The architecture organizes these traces through two distinct, non-generative structural views: an entity-context graph to map connections across different interactions and a temporal hierarchy to maintain conversational locality and session state. When a query is issued, the system performs a coordinated retrieval from both views, using deterministic calibration to resolve conflicting evidence and ensure the final response remains grounded in the raw traces. Crucially, the LLM is only invoked during the final question-answering stage; all memory operations bypass LLM input and output token consumption. Experimental results on long-memory and long-context question-answering benchmarks demonstrate that Zero-Mem achieves performance competitive with existing methods while significantly improving efficiency. Specifically, when utilizing the same final-QA reader and context budget, Zero-Mem reduces memory-operation time costs by 57.6% compared to the fastest baseline, proving that structured agent memory can function effectively without the need for generative intermediate representations.