The Marigold V2 framework introduces an advanced approach to monocular depth estimation by repurposing Diffusion Transformer (DiT) architectures, originally designed for image generation, for dense regression tasks. To achieve efficient single-step inference, the researchers utilize pretrained multi-step flow-matching models and incorporate quantization to maintain high model capacity while reducing computational costs. To mitigate artifacts common in naive training, the authors implement two primary technical remedies: the alignment of internal model representations with semantic features derived from ground-truth data and a specialized two-stage fine-tuning protocol centered on a novel Sinkhorn-based loss function. This methodology enables the model to resolve fine-grained geometric details, such as hair-thin edges, foliage, and fur, which typically pose challenges for existing estimators. Quantitatively, Marigold V2 demonstrates superior generalization to out-of-distribution inputs, yielding a significant 16-26% improvement in Absolute Relative Error (AbsRel) over previous benchmarks on the KITTI and ETH3D datasets. Beyond depth estimation, the architecture proves highly effective for other dense regression applications, including surface normal estimation and intrinsic image decomposition, establishing a new state-of-the-art performance level for transformer-based generative models in computer vision tasks.
Read original
huggingface/daily-papers