Researchers have introduced DramaChain Bench, a comprehensive end-to-end evaluation framework designed to assess the multi-stage production pipeline of commercial short dramas, including scriptwriting, storyboarding, keyframe generation, shot-level video, and final assembly. Unlike previous benchmarks that focus exclusively on video generation, this framework evaluates the entire chain to determine if downstream outputs maintain the original script's intent and exhibit multi-episode coherence. The benchmark utilizes the DramaChain Dimensions system, which consists of five high-level evaluation axes that expand into 63 specific leaf dimensions. To ensure high-quality ground truth, the DramaChain Labeling System employed three professional annotators to score 5,785 items, generating 17,488 valid scores and 255,925 traceable attribution records with spatio-temporal defect localization. Furthermore, the authors developed the DramaChain Agentic Judge, an automated scoring mechanism that utilizes multiple agentic rounds and per-item checklists to evaluate leaf dimensions. This automated judge demonstrated high reliability, achieving a Pearson Linear Correlation Coefficient (PLCC) of 0.918 compared to human annotations. This high correlation suggests that the agentic judge can effectively rank new models without the need for expensive manual annotation, while simultaneously confirming that upstream errors in the production chain cascade through to the final video quality.
Read original
huggingface/daily-papers