Researchers introduced ModaLens, an image-swap audit framework designed to evaluate image sensitivity in report-conditioned medical vision-language models (VLMs). The study investigated whether models actually utilize visual input or rely solely on textual radiology reports to answer clinical questions. Using MedGemma-27B, the authors analyzed 3,199 paired MIMIC-CXR cases from 293 patients, employing 14 distinct questions per case—comprising 13 finding-specific queries and one composite query. The methodology involved replacing the original image with another from the same patient while keeping the report and question constant to measure sensitivity to visual changes. Results demonstrated that the availability of a radiology report significantly reduces image-swap sensitivity. Under explicit answer instructions, the model's output changed in only 4.26% of trials when the report was provided, compared to 20.94% when the report was absent, representing a paired increase in sensitivity of 16.7 percentage points (95% CI 15.6 to 17.7). When using the original prompt with a lowercase first-token readout, sensitivity shifted from 4.70% with the report to 17.07% without it. Furthermore, the study found that image substitutions influenced continuous answer scores even when binary predictions remained static. These findings, which were replicated across two additional model lineages, suggest that report availability mitigates the model's reliance on visual features during clinical question answering.

Read original