The paper introduces online, instruction-conditioned mistake detection for egocentric procedural videos, enabling real-time assessment of action attempts against task instructions. The model processes observed video frames, instruction text, and prior predictions to identify and temporally localize completed steps, classifying them as correct, modified, slip, correction, or addition; omitted steps are inferred post-video. Two architectures are compared: a modular pipeline with separate step identification, temporal localization, and execution verification stages, and a joint model predicting all outputs in a single forward pass. To enable supervised training on erroneous executions, the EgoPER dataset is repartitioned to include mistake-containing videos across all splits. Both models achieve comparable step-segmentation F1@0.5 (93.6% modular, 92.1% joint), but the joint model outperforms on binary mistake detection F1@0.5 (84.1% vs. 80.7%), with the largest improvement on slip detection (56.4% vs. 37.5%). Component-level analysis attributes the modular approach's lower detection performance to error propagation between stages. On test videos with mistakes, both designs surpass the released AEM and ESTANet checkpoints under the proposed evaluation protocol.
Read original
huggingface/daily-papers