US medical licensing exam style questions. The original medical LLM benchmark and still the most cited.
Audited by MedCheckJin et al., Applied Sciences 2021
Key facts
| What it measures | US medical licensing exam style questions. The original medical LLM benchmark and still the most cited. |
| Who should care | Researchers comparing to prior literature. |
| Use it when | You need comparability with years of published results. |
| Do not use it for | Predicting clinical performance. Exam scores correlate poorly with real clinical utility, and contamination risk is high after years of reuse. |
| Score scale | Percent accuracy, multiple choice. |
| Grader method | unknown |
| First released | 2021 |
| Languages | English, Chinese |
| Use cases | exam knowledge |
| Citation | Jin et al., Applied Sciences 2021 |
| Status | active |
Frequently asked questions
What does MedQA measure?
US medical licensing exam style questions. The original medical LLM benchmark and still the most cited.
Who should use MedQA?
Researchers comparing to prior literature. You need comparability with years of published results.
What should MedQA not be used for?
Predicting clinical performance. Exam scores correlate poorly with real clinical utility, and contamination risk is high after years of reuse.
How are MedQA scores reported?
Percent accuracy, multiple choice. Scores from different benchmarks are not comparable to each other.
Is MedQA independent?
No conflicts of interest have been verified for MedQA in this registry as of 2026-08-05. Absence of a recorded conflict means none has been verified, not that none exists.
Compare with
Multi-specialty medical questions written by clinicians and students across African countries, built to test w
Chinese-language evaluation of physical and mental health knowledge in large models.
Comprehensive Chinese medical benchmark spanning exam knowledge and clinical case work.
Chinese medical licensing exam questions with expert annotations for reasoning and difficulty.
Evaluation framework and dataset covering several medical specialties, built around clinician-style diagnostic
Spanish healthcare specialization exam questions requiring multi-step reasoning.
