Researchers propose prepending trainable [DUMP] prefix tokens to input sequences during embedding model training to mitigate representational interference caused by residual token-level artifacts, such as attention sink behavior, in fixed-size retrieval vectors. The method preserves the original model architecture, pooling strategy, and contrastive retrieval loss, optimizing only the added prefix tokens through standard training objectives. By introducing learnable positions that interact with regular tokens during encoding, [DUMP] tokens absorb low-information signals, reducing their influence on the final pooled representation. Experiments across standard retrieval and long-context benchmarks demonstrate consistent improvements over fine-tuned baselines for both encoder and decoder embedding models. Analysis of token-level activations and attention patterns confirms that the prefix tokens alter the dynamics associated with uninformative tokens, offering a lightweight architectural modification that enhances retrieval representation quality without increasing inference complexity or altering the core training pipeline.