Whether a model refuses or complies with medically harmful requests.
Audited by MedCheckHan et al., NAACL Industry Track 2024
Key facts
| What it measures | Whether a model refuses or complies with medically harmful requests. |
| Who should care | Anyone deploying a patient-facing tool. |
| Use it when | Safety review before launch. |
| Do not use it for | Measuring clinical accuracy. |
| Score scale | Harmfulness score across harmful request categories. |
| Grader method | unknown |
| First released | 2024 |
| Languages | English |
| Use cases | safety |
| Citation | Han et al., NAACL Industry Track 2024 |
| Status | active |
Frequently asked questions
What does MedSafetyBench measure?
Whether a model refuses or complies with medically harmful requests.
Who should use MedSafetyBench?
Anyone deploying a patient-facing tool. Safety review before launch.
What should MedSafetyBench not be used for?
Measuring clinical accuracy.
How are MedSafetyBench scores reported?
Harmfulness score across harmful request categories. Scores from different benchmarks are not comparable to each other.
Is MedSafetyBench independent?
No conflicts of interest have been verified for MedSafetyBench in this registry as of 2026-08-05. Absence of a recorded conflict means none has been verified, not that none exists.
Compare with
Physician-written rubrics score open-ended answers to realistic health conversations, including patient-facing
Whether a model computes clinical scores and formulas correctly from a patient note.
Whether a model can find and fix clinical errors already present in a note.
A framework covering several clinical application dimensions at once, including safety and reasoning, rather t
Whether a model asks the right follow-up question when it does not have enough information, instead of guessin
Quantifies reasoning quality on real-world clinical cases, scoring the path rather than only the answer.
