Youtu-Parsing-Omni introduces a unified omni-modal parser that converts documents, images, audio, and video into space-time grounded structured records using a single encoder and schema. Its core, the Youtu-Omni-Encoder, initializes from a pre-trained language model to enhance text reading and encodes all modalities with shared weights within a unified space-time coordinate system; a few fusion layers enable video frames to attend to concurrent audio, integrating cross-modal fusion and temporal alignment into the perception pipeline. OmniSchema consolidates outputs from ten parsing tasks into a single self-evolving post-training framework, OmniSchema-Aware On-Policy Distillation (OSAD), which optimizes a shared token-role-based objective. In each training round, the model learns against token-level targets from a reference-conditioned copy of itself, whose weights are then refreshed from the trained model, enabling the parser and its supervision to co-evolve. With a 5-billion-parameter checkpoint, the system achieves state-of-the-art Overall score of 96.96 on OmniDocBench v1.6 and leads all open-weight models with an average score of 75.08 on OmniParsingBench, with performance gains concentrated on audio and video modalities. It also sets a new benchmark of 95.02 on overall chart parsing and achieves the highest exact chemical-fingerprint agreement, while matching a dedicated optical-music model in typeset sheet music transcription and outperforming general-purpose systems.
Read original
huggingface/daily-papers