The FLAT (Flexible-Length Aligned Transmodal representations) framework introduces a unified pre-training approach designed to bridge the gap between multimodal representation learning and generative modeling. Unlike traditional two-stage methods that rely on frozen embeddings, FLAT jointly optimizes a shared multimodal encoder with text-to-image (T2I) and image-to-text (I2T) decoders. This architecture maps visual and textual inputs into a unified, continuous 1D sequence space. To facilitate dynamic output lengths and flexible-length tokenization, the researchers implemented nested dropout over prefix-K tokens. By combining contrastive alignment with bidirectional cross-modal generative objectives, the model produces linearly interpolatable embeddings capable of serving as both discriminative descriptors and generative conditions. During pre-training, FLAT achieved a GenEval score of 71.1 for T2I generation. Following task-specific fine-tuning, the model reached a GenEval score of 83.1 for T2I, alongside 40.5 BLEU-4 and 138.6 CIDEr on MS-COCO image captioning. Retrieval performance was robust, with Recall@5 reaching 86.8 for I2T and 75.8 for T2I on MS-COCO, and even higher values on Flickr30K (98.3 I2T / 93.6 T2I). Qualitatively, the embeddings natively support latent space arithmetic, linear interpolation, and zero-shot composed retrieval.
Read original
huggingface/daily-papers