StableVQ introduces a novel framework designed to improve the training stability and performance of vector-quantized (VQ) tokenizers, which are essential for autoregressive and masked image generation models. The authors identify the primary cause of training instability as the entanglement between the Encoder-Decoder and the Codebook modules, where each subsystem fails to fulfill its role independently. To resolve this, StableVQ implements three key technical innovations built upon existing shared-projection codebook architectures without adding new learnable parameters. First, Dynamic STE (Straight-Through Estimator) stabilizes the Encoder's learning objective, allowing it to optimize the reconstruction space effectively even during periods of low codebook utilization. Second, the proposed Region VQ Loss redefines the Codebook's objective, ensuring it can track the encoder output distribution independently of encoder oscillations. Finally, a Decoupled Schedule assigns distinct learning rate schedules to the Encoder-Decoder and the Codebook to accommodate their differing optimization dynamics. Extensive experiments conducted on the ImageNet dataset demonstrate that StableVQ consistently enhances training stability, increases codebook utilization, and improves overall reconstruction quality across various codebook sizes and initialization configurations, providing a more robust approach to training discrete visual tokenizers.

Read original