Real-world clinical benchmark with physician validation of both questions and reference answers.
Audited by MedCheckZhang et al., 2025
Key facts
| What it measures | Real-world clinical benchmark with physician validation of both questions and reference answers. |
| Who should care | Teams that want clinician-validated ground truth rather than scraped exam keys. |
| Use it when | You care that a doctor checked the answer key. |
| Do not use it for | Large-scale automated comparison. Physician validation limits the size. |
| Score scale | Physician-validated scoring on open responses. |
| Grader method | unknown |
| First released | 2025 |
| Languages | Chinese, English |
| Use cases | clinical decision support |
| Citation | Zhang et al., 2025 |
| Status | active |
Frequently asked questions
What does LLMEval-Med measure?
Real-world clinical benchmark with physician validation of both questions and reference answers.
Who should use LLMEval-Med?
Teams that want clinician-validated ground truth rather than scraped exam keys. You care that a doctor checked the answer key.
What should LLMEval-Med not be used for?
Large-scale automated comparison. Physician validation limits the size.
How are LLMEval-Med scores reported?
Physician-validated scoring on open responses. Scores from different benchmarks are not comparable to each other.
Is LLMEval-Med independent?
No conflicts of interest have been verified for LLMEval-Med in this registry as of 2026-08-05. Absence of a recorded conflict means none has been verified, not that none exists.
Compare with
Simulated clinical encounters where the model must gather information over several turns rather than answer on
Real clinical practice text, drawn from actual notes rather than exam material, across many languages and task
Large-scale Chinese benchmark built around real clinical scenarios rather than exam questions.
Broad clinical benchmark covering summarization, diagnosis and treatment planning in one suite.
Comprehensive Chinese medical benchmark spanning exam knowledge and clinical case work.
Questions answered from real discharge summaries, written by clinicians for real-world practice.
