Framework
Clinical evaluation principles and how this kit applies them.
NOHARM is a clinical AI evaluation framework developed by the ARISE Network. It evaluates AI performance in healthcare based on benefit and harm using established clinical standards.
Safety is inseparable from clinical performance because errors can cause irreversible harm.
In healthcare, performance depends on the severity of errors, not accuracy alone. Just as autonomous vehicle evaluation must account for crash rates, medical AI evaluation must account for the harm of its errors.
We urge all medical AI benchmark designers to follow these principles.
Applications
The initial NOHARM study evaluated AI performance for clinicians asking about realistic patient cases. The framework, however, is broadly applicable and can be adapted to any clinical AI evaluation, including multi-turn, multimodal, and agentic benchmarks.
NOHARM core principles
An unsafe doctor is not a good doctor. AI performance must be evaluated the same way, with safety designed into benchmarks from the start.
A medical error can cause minimal symptoms or be life-threatening. Benchmarks that ignore harm severity miss what matters to clinicians and patients and mismeasure AI performance.
Even detailed numeric rubric scores need clinical meaning. Clinical appropriateness and harm severity, such as '10% of responses could cause severe harm,' are immediately interpretable and meaningful to clinicians and patients.
Clinical rubrics and measurement
Rubrics make clinical judgments explicit: what counts as appropriate care, which errors matter, and how severe their potential consequences are.
Clinical appropriateness
Judge whether an action's expected benefits outweigh its risks for this patient, using clinical evidence and expert judgment.
- Inappropriate
- Uncertain
- Appropriate
Potential harm
Use explicit severity definitions to rate potential harm from incorrect actions and missed necessary care.
- None
- Mild
- Moderate
- Severe
- Death
NOHARM is agnostic to the framework or taxonomy. Choose an established, validated clinical framework or taxonomy suited to your setting rather than inventing one.
Interpretable results
Report clinical appropriateness and the frequency of each harm severity, with the denominator and rating method.
Illustrative result: 10% of evaluated responses contained an error that could cause severe harm.