The InclusionMed Collaboration introduces InclusionMed, an open evaluation initiative for inclusive medical AI organized around the InclusionMed Atlas, which structures benchmarks across three healthcare roles and three AI capabilities. For the initial release, four public benchmarks were selected from 388 candidates via structured audits and independent reproduction, complemented by InclusionMed-DxFiles, a private diagnostic reasoning benchmark, yielding five benchmarks covering three of nine role-capability cells. Frontier large language models were evaluated on this Atlas and aggregated into the InclusionMed Index, producing overall scores ranging from 17.5 to 49.1 out of 100. The Index further reports scores disaggregated by healthcare role and AI capability, revealing that model rankings shift depending on the specific role or capability assessed. A within-benchmark analysis using task-level tags demonstrated that the ten highest-ranked models consistently scored higher on patient communication tasks than on clinical decision-making tasks. Additionally, models with comparable Index scores exhibited substantial differences in estimated inference cost. These results underscore the necessity for evaluation frameworks spanning diverse healthcare needs and the importance of context-specific model selection. InclusionMed aims to expand coverage through open collaboration and community contributions to ensure global healthcare needs shape medical AI progress measurement.
Read original
huggingface/daily-papers