To prevent train/test contamination from inflating model benchmarks, developers should implement a three-tiered auditing process during continuous integration. The first level is an exact join using hash lookups to identify byte-for-byte matches. The second level involves normalization—applying Unicode NFKC, lowercasing, and stripping punctuation—to catch reformatted text. The third and most complex level is near-duplicate detection, which utilizes n-gram shingling (typically 5-grams) and MinHash with Locality Sensitive Hashing (LSH) to approximate the Jaccard index. To maintain efficiency, the Jaccard similarity should be calculated directly from the shingle sets of LSH-generated candidates rather than from the MinHash signatures themselves. When reporting results, developers should disclose contamination rates at each level (exact, normalized, and near-duplicate) as a fraction of the total evaluation records to ensure defensibility. If overlap is detected, the correct mitigation is to remove the offending records from the training set and retrain, rather than modifying the evaluation set, which preserves the integrity of historical comparisons. While these methods cannot detect semantic paraphrasing or contamination embedded in a base model's weights, they effectively identify common leaks caused by dataset merging, augmentation-induced splits, or re-scraping.
Read original
dev.to