Dashboard

Pipeline

DatasetImport inputs and references1 / 1items imported RubricsDraft and review Decisions and Options1 / 1items with rubrics0 of 1 reviewed ResponsesCollect model responses3 / 3responses collected AutogradingMap model responses onto the rubric3 / 3responses autograded0 of 3 reviewed ScoresCompute scores and harm3 / 3responses scored0 of 3 reviewed LeaderboardCompare models across items3 / 3models with scores0 of 3 reviewed

Review and issues

Rubric reviewSelected or previously used rubrics not yet reviewed1

Customize

Shape the benchmark around your question. Discuss these choices with your coding agent, then agree on the setup before running it.

Benchmark design choices and alternatives
Decision	Option	Alternative
RAND review	Quick One independent rating round for an initial review.	Full Two rating rounds with discussion and a moderator. Requires more model calls.
RAND panel size	Three members The minimum supported panel, with fewer model calls.	Larger panel Include more model raters, at a higher cost per review.
RAND panel models	Different model families Compare ratings across models with different training and behavior.	Repeated model Check a model's consistency across independent calls, rather than diversity across models.
Models to benchmark	Focused comparison Test a small set of models relevant to your intended use.	Broader comparison Include a range of model families and capabilities to explore differences in performance.
Severity weights	Toolkit weights Uncertain 1, mild 3, moderate 24, severe 72. More severe harms contribute more to the score.	Custom weights Use a different increasing scale for a separate comparison. Requires a scoring-code change and new results.
Responses per item	One response per model A smaller run, with one response for each item and model.	Repeated responses Measure variation across multiple model responses. Choose the count before collecting responses.
Autograder	One fixed judge Use the same model and reasoning setting to map all model responses onto the rubric.	Compare judges Grade the same model responses in separate runs to examine how the judge affects results.
Clinical review	Automated drafts Explore the pipeline with provisional rubrics and results.	Clinician review Review the rubric and grading evidence against your declared clinical review requirements.

Ask your agent to inspect your current setup before making changes. RAND model panels provide advisory ratings; model agreement does not establish clinical validation. Preserve completed results when planning a new run.

Discuss the setup with your coding agent:

Help me customize demo.

Dashboard

Customize

Shape the benchmark around your question. Discuss these choices with your coding agent, then agree on the setup before running it.

Benchmark design choices and alternatives
Decision	Option	Alternative
RAND review	Quick One independent rating round for an initial review.	Full Two rating rounds with discussion and a moderator. Requires more model calls.
RAND panel size	Three members The minimum supported panel, with fewer model calls.	Larger panel Include more model raters, at a higher cost per review.
RAND panel models	Different model families Compare ratings across models with different training and behavior.	Repeated model Check a model's consistency across independent calls, rather than diversity across models.
Models to benchmark	Focused comparison Test a small set of models relevant to your intended use.	Broader comparison Include a range of model families and capabilities to explore differences in performance.
Severity weights	Toolkit weights Uncertain 1, mild 3, moderate 24, severe 72. More severe harms contribute more to the score.	Custom weights Use a different increasing scale for a separate comparison. Requires a scoring-code change and new results.
Responses per item	One response per model A smaller run, with one response for each item and model.	Repeated responses Measure variation across multiple model responses. Choose the count before collecting responses.
Autograder	One fixed judge Use the same model and reasoning setting to map all model responses onto the rubric.	Compare judges Grade the same model responses in separate runs to examine how the judge affects results.
Clinical review	Automated drafts Explore the pipeline with provisional rubrics and results.	Clinician review Review the rubric and grading evidence against your declared clinical review requirements.

Discuss the setup with your coding agent:

Help me customize demo.