The paper introduces PixSDS, a method designed to mitigate the structured color artifacts and high-frequency texture noise commonly produced by latent Score Distillation Sampling (SDS) in text-to-3D generation. The authors identify the root cause as "VAE-induced pixel drift," a phenomenon where the optimization process moves the image along pixel-space directions that are weakly constrained by the Variational Autoencoder (VAE) encoder. This allows the latent representation to appear semantically consistent while the corresponding pixel-space image accumulates significant noise due to the underconstrained inverse mapping. To address this, the researchers propose PixSDS, a lightweight gradient repair technique that maintains the original SDS objective and diffusion model. PixSDS functions by decoding a latent SDS lookahead step into pixel space, utilizing the resulting decoded image as a clean directional guide for optimization. This approach minimizes motion in VAE-inconsistent directions. Through controlled 2D SDS experiments and 3D text-to-3D generation tasks, the authors demonstrate that PixSDS effectively reduces structured artifacts and noise while successfully preserving the intended semantic content of the generated objects without requiring model retraining or changes to the rendering pipeline.
Read original
huggingface/daily-papers