Questions answered from real discharge summaries, written by clinicians for real-world practice.
Audited by MedCheckKweon et al., NeurIPS 2024
Key facts
| What it measures | Questions answered from real discharge summaries, written by clinicians for real-world practice. |
| Who should care | Health systems evaluating record summarization and chart question answering. |
| Use it when | Testing whether a model can actually read a discharge summary. |
| Do not use it for | Open-ended conversation quality. |
| Score scale | Accuracy against clinician-written reference answers. |
| Grader method | unknown |
| First released | 2024 |
| Languages | English |
| Use cases | documentation, clinical decision support |
| Citation | Kweon et al., NeurIPS 2024 |
| Status | active |
Frequently asked questions
What does EHRNoteQA measure?
Questions answered from real discharge summaries, written by clinicians for real-world practice.
Who should use EHRNoteQA?
Health systems evaluating record summarization and chart question answering. Testing whether a model can actually read a discharge summary.
What should EHRNoteQA not be used for?
Open-ended conversation quality.
How are EHRNoteQA scores reported?
Accuracy against clinician-written reference answers. Scores from different benchmarks are not comparable to each other.
Is EHRNoteQA independent?
No conflicts of interest have been verified for EHRNoteQA in this registry as of 2026-08-05. Absence of a recorded conflict means none has been verified, not that none exists.
Compare with
Real clinical practice text, drawn from actual notes rather than exam material, across many languages and task
Broad clinical benchmark covering summarization, diagnosis and treatment planning in one suite.
A realistic virtual electronic health record environment where agents must complete tasks by taking actions, n
Whether a model understands medical coding vocabularies such as ICD and ATC, including rare and easily confuse
Whether a model can find and fix clinical errors already present in a note.
Tests models on 121 real clinical work tasks, not exam questions: summarizing records, drafting discharge inst
