This research introduces Visual Prompt Engineering (VIPE), a novel technique designed to enhance the performance of video foundation models in visual reasoning tasks. Unlike traditional text-based prompt engineering, VIPE focuses on automatically modifying the input task image to optimize model response. For instance, the method can transform abstract, sketch-like scenes into photorealistic environments using image editing models, thereby providing more effective visual cues for the video model. The authors demonstrate that VIPE significantly improves reasoning capabilities across various tasks, such as predicting physical outcomes like where an object lands after navigating obstacles. Notably, the study finds that visual prompt engineering can outperform both classic text-based prompting and test-time scaling methods for video models. By treating the visual input as a tunable prompt, VIPE offers a compute-efficient alternative to increasing model scale or inference time, serving as a systematic approach to elicit superior reasoning performance from existing video foundation models. This approach suggests that the visual content itself can be engineered to bridge the gap between complex task requirements and the model's internal representations.
Read original
huggingface/daily-papers