Medical questions paired with multiple valid expert explanations, so a model is judged on reasoning as well as the answer.
Audited by MedCheckKim et al., BioNLP 2024
Key facts
| What it measures | Medical questions paired with multiple valid expert explanations, so a model is judged on reasoning as well as the answer. |
| Who should care | Teams that need explainable output, not just correct output. |
| Use it when | Evaluating explanation quality alongside correctness. |
| Do not use it for | Speed comparisons. Explanation scoring is slower. |
| Score scale | Accuracy plus explanation quality scoring. |
| Grader method | unknown |
| First released | 2024 |
| Languages | English |
| Use cases | clinical decision support, patient communication |
| Citation | Kim et al., BioNLP 2024 |
| Status | active |
Frequently asked questions
What does MedExQA measure?
Medical questions paired with multiple valid expert explanations, so a model is judged on reasoning as well as the answer.
Who should use MedExQA?
Teams that need explainable output, not just correct output. Evaluating explanation quality alongside correctness.
What should MedExQA not be used for?
Speed comparisons. Explanation scoring is slower.
How are MedExQA scores reported?
Accuracy plus explanation quality scoring. Scores from different benchmarks are not comparable to each other.
Is MedExQA independent?
No conflicts of interest have been verified for MedExQA in this registry as of 2026-08-05. Absence of a recorded conflict means none has been verified, not that none exists.
Compare with
Simulated clinical encounters where the model must gather information over several turns rather than answer on
Real clinical practice text, drawn from actual notes rather than exam material, across many languages and task
Large-scale Chinese benchmark built around real clinical scenarios rather than exam questions.
Broad clinical benchmark covering summarization, diagnosis and treatment planning in one suite.
Comprehensive Chinese medical benchmark spanning exam knowledge and clinical case work.
Questions answered from real discharge summaries, written by clinicians for real-world practice.
