Expert-level medical reasoning questions deliberately built to be harder than licensing exams, with a multimodal track alongside the text track.
Zuo et al., 2025
Key facts
| What it measures | Expert-level medical reasoning questions deliberately built to be harder than licensing exams, with a multimodal track alongside the text track. |
| Who should care | Teams whose models already score near the ceiling on MedQA and need headroom. |
| Use it when | MedQA and MedMCQA have saturated and you need a benchmark that still discriminates. |
| Do not use it for | Comparability with the older literature. It is intentionally not the same difficulty. |
| Score scale | Percent accuracy, text and multimodal tracks reported separately. |
| Grader method | multiple choice |
| First released | 2025 |
| Languages | English |
| Use cases | exam knowledge, clinical decision support, multimodal |
| Citation | Zuo et al., 2025 |
| Status | active |
Frequently asked questions
What does MedXpertQA measure?
Expert-level medical reasoning questions deliberately built to be harder than licensing exams, with a multimodal track alongside the text track.
Who should use MedXpertQA?
Teams whose models already score near the ceiling on MedQA and need headroom. MedQA and MedMCQA have saturated and you need a benchmark that still discriminates.
What should MedXpertQA not be used for?
Comparability with the older literature. It is intentionally not the same difficulty.
How are MedXpertQA scores reported?
Percent accuracy, text and multimodal tracks reported separately. Scores from different benchmarks are not comparable to each other.
Is MedXpertQA independent?
No conflicts of interest have been verified for MedXpertQA in this registry as of 2026-08-05. Absence of a recorded conflict means none has been verified, not that none exists.
Compare with
Multi-specialty medical questions written by clinicians and students across African countries, built to test w
Chinese-language evaluation of physical and mental health knowledge in large models.
Comprehensive Chinese medical benchmark spanning exam knowledge and clinical case work.
Chinese medical licensing exam questions with expert annotations for reasoning and difficulty.
Evaluation framework and dataset covering several medical specialties, built around clinician-style diagnostic
Spanish healthcare specialization exam questions requiring multi-step reasoning.
