Quick Reference

From your dataset to a model leaderboard.

Welcome to NOHARMKit

Build severity-weighted AI benchmarks from your own data, using the NOHARM framework. This kit is currently streamlined for a single-turn benchmark, but you can adapt the framework to any multi-turn, multimodal, or agentic evaluation.

Create your project and prepare your dataset here. Copy each step's prompt into Codex or Claude Code to run the pipeline, then return here to inspect and review the results. No agent connection is required.

NOHARM framework

Review the core principles before building your benchmark.

Read the Framework View Tutorial

Pipeline Overview

1
Dataset
Enter cases or upload your dataset.
2
Rubrics
Draft and review Decisions and Options.
3
Responses
Collect model responses from selected models.
4
Autograding
Map model responses to the chosen rubric.
5
Scores
Compute adequacy and harm.
6
Leaderboard
Compare models across items.

Your first benchmark

1
Create a project
Choose New project and name your benchmark.
2
Add your dataset
Enter Task, Solution, and Errors, or use Import dataset.
3
Generate and review
Copy the handoff prompt into your coding agent, then return to Rubrics.

Create a New Project

Stage buttons open prompts for you to copy. They do not run model calls. After your agent finishes, reload the page to see its output, then use Next to move to the next stage.

Open your agent in this repository. The skills ship with the kit; start a new session if they are missing.
Use 1-64 lowercase letters, digits, or hyphens, starting with a letter or digit.
Choose model routes and authorize paid calls before generation, responses, or autograding.
Copying a prompt does not run it. Browser selection does not select a project for your agent; the prompts name it explicitly.

If you have not launched the app yet, follow First launch in the repository README. It includes the setup prompt.
If a skill is missing, start a new agent session in the whole repository folder, the one containing README.md, tools, and app.
If your project or output is missing, refresh the app and check the selected project. Ask your agent to confirm it wrote to the project named in the prompt.
If a step fails, keep its output and ask the agent to inspect it. Use the continuation prompt below to find the next missing, failed, or stale step.

Pipeline

For your own project, select each step to see what to supply, what the agent produces, and how to check it. Generation, model responses, and autograding may need provider setup and paid calls; agree on the models and run scope with your agent first.

Create and select your own project before running these prompts. Until then, replace my-benchmark with your chosen project name.

Rubrics and model responses meet in autograding. Scores and the leaderboard use those stored judgments.

Dataset

Open Dataset

Enter a Task and Solution in Dataset, or use Import dataset to upload a CSV or JSON file. Add acceptable alternatives and possible Errors, then copy the handoff prompt into your coding agent.

You supply: A Task, one or more acceptable Solutions, and optional Errors with harm levels.
Expected output: Your upload is saved in the project's imports folder. Your coding agent processes it into cases and drafts rubrics.
Before continuing: Check that Task, Solution, and Errors match your source. Solutions are acceptable alternatives, not cumulative requirements.

Prompt for my-benchmark:

Use noharm-import-cases for project "my-benchmark". Import my supplied dataset.

Upload a CSV or JSON file with Task, Solution, and Errors fields. Task and Solution must be nonempty; Errors are optional but recommended.

For CSV, put multiple acceptable solutions in one Solution field, separated by SOLUTION:. JSON can use an array of solutions:

[
  {
    "Task": "Your task",
    "Solution": [
      "An acceptable solution",
      "Another acceptable solution"
    ],
    "Errors": "Possible errors and their potential severity"
  }
]

Import dataset saves the original file in your project's imports folder. Paste its handoff prompt into your coding agent to process it and generate rubrics. Supplied Errors and harm levels remain author-provided criteria.

Cases without supplied IDs receive generic names such as Case-001 and Case-002. Their stored IDs are Case001 and Case002. Supplied IDs are preserved; incompatible IDs need an agreed mapping. Stored IDs cannot contain hyphens or path separators, or start with Ex.

For direct agent imports, the legacy JSON keys remain presentation for Task and recommendations for Solution. Both must be nonempty strings.

Rubrics

Open Rubrics

A rubric turns your Solution and supplied Errors into Decisions and Options. Open the rubric editor from the Output card to review and edit the draft without leaving NOHARMKit.

You supply: Imported items and any existing criteria or ratings you want the agent to use.
Expected output: A rubric version with Decisions, appropriate and inappropriate Options, and validation results.
Before continuing: Use Open Rubric Editor on the Output card. Check Decisions, Options, appropriateness, and harm levels; save, then close the editor. Choose the saved version before autograding.

Prompt for my-benchmark:

Use noharm-rubrics for project "my-benchmark". Draft rubrics for my imported items and help me review them.

Responses

Open Responses

Choose models in the Models to test checklist, then use Copy model selection to give the plan to your agent. Collect Responses opens a separate prompt to run the agreed models. Models see the Task without the Solution or Errors.

You supply: Use the checklist and Copy model selection, or give your agent the full model list. OpenRouter supports testing models from multiple organizations.
Expected output: One stored response for each expected case, model, and declared trial.
Before continuing: Check missing and failed responses. Decide whether to retry them before comparing models.

Prompt for my-benchmark:

Use noharm-responses for project "my-benchmark". Collect responses using the agreed models and declared trials.

Autograding

Open Autograding

The judge compares each stored model response with your chosen rubric and maps it to Decisions and Options. Confirm the rubric version and judge model, then inspect the mappings and failures.

You supply: Stored model responses, the rubric version to use, and your chosen judge model.
Expected output: Mappings from each model response to rubric Options, with quoted evidence.
Before continuing: Inspect a complete and an incomplete model response. Investigate failed or out-of-date mappings before scoring.

Prompt for my-benchmark:

Use noharm-autograding for project "my-benchmark". Grade the stored responses; confirm the rubric version and judge model with me.

The default judge is GPT-6.1 Sol on OpenRouter with high reasoning effort. Confirm the rubric version, route, and spend before running. Autograding covers the project's stored responses across declared trials. Existing judgments are preserved; changed rubrics or rejudging require a fresh project/run. For interrupted direct OpenAI runs, verified resume can run only missing jobs with unchanged inputs and configuration.

Scores

Open Scores

Scores are calculated deterministically from the stored judgments. Review Recall, Precision, F1, harm, and coverage as separate measures.

You supply: The stored judgments for the intended rubric version. This step needs no model call.
Expected output: Recall, Precision, F1, and harm for each scored model response, with coverage information.
Before continuing: Trace unexpected scores back to the model response and mapping. Missing model responses are not zero scores.

Prompt for my-benchmark:

Use noharm-scores for project "my-benchmark". Compute scores and check coverage.

Leaderboard

Open Leaderboard

The leaderboard ranks models by Mean F1 across their available scored responses, including partial and unverified results. Compare adequacy and harm alongside coverage and review status.

You supply: Scores already stored in this project. There is no separate generation step.
Expected output: A model comparison across the project's items.
Before continuing: Check coverage, harm, and review status alongside F1. A draft comparison is not a clinically validated result.

Results and review

Adequacy and harm describe different parts of model performance. Read them alongside coverage and review status to understand what the results support.

Adequacy: Recall, Precision, and F1
Safety: Harm, reported separately
Evidence: Coverage and review status

After a rubric edit

Save reviewed copy
Prepare fresh project/run
Autograde intended rubric
Recompute scores

LLM draft means model-produced. Validation is not clinical review.
Provisional means the evidence has limits; an average does not remove them.
Missing or failed responses are not zero scores. Check coverage before comparing models.
Saving a reviewed rubric does not update existing judgments. Preserve earlier outputs and use a fresh project/run for rejudging. Verified resume for an interrupted direct OpenAI run can fill missing jobs only; it does not replace failed judgments or accept changed rubrics.
A completed pipeline run and a completed review are separate.

Continuing a project

Continue with:

Use noharm for project "my-benchmark". Inspect the stored outputs and help me run the next pipeline step.

Welcome to NOHARMKit

Your first benchmark

Create a project

Choose New project and name your benchmark.

Add your dataset

Enter Task, Solution, and Errors, or use Import dataset.

Generate and review

Copy the handoff prompt into your coding agent, then return to Rubrics.

Create a New Project

Stage buttons open prompts for you to copy. They do not run model calls. After your agent finishes, reload the page to see its output, then use Next to move to the next stage.

Open your agent in this repository. The skills ship with the kit; start a new session if they are missing.
Use 1-64 lowercase letters, digits, or hyphens, starting with a letter or digit.
Choose model routes and authorize paid calls before generation, responses, or autograding.
Copying a prompt does not run it. Browser selection does not select a project for your agent; the prompts name it explicitly.

If you have not launched the app yet, follow First launch in the repository README. It includes the setup prompt.
If a skill is missing, start a new agent session in the whole repository folder, the one containing README.md, tools, and app.
If your project or output is missing, refresh the app and check the selected project. Ask your agent to confirm it wrote to the project named in the prompt.
If a step fails, keep its output and ask the agent to inspect it. Use the continuation prompt below to find the next missing, failed, or stale step.

Pipeline

Create and select your own project before running these prompts. Until then, replace my-benchmark with your chosen project name.

Rubrics and model responses meet in autograding. Scores and the leaderboard use those stored judgments.

Dataset

Open Dataset

Enter a Task and Solution in Dataset, or use Import dataset to upload a CSV or JSON file. Add acceptable alternatives and possible Errors, then copy the handoff prompt into your coding agent.

You supply: A Task, one or more acceptable Solutions, and optional Errors with harm levels.
Expected output: Your upload is saved in the project's imports folder. Your coding agent processes it into cases and drafts rubrics.
Before continuing: Check that Task, Solution, and Errors match your source. Solutions are acceptable alternatives, not cumulative requirements.

Prompt for my-benchmark:

Use noharm-import-cases for project "my-benchmark". Import my supplied dataset.

Upload a CSV or JSON file with Task, Solution, and Errors fields. Task and Solution must be nonempty; Errors are optional but recommended.

For CSV, put multiple acceptable solutions in one Solution field, separated by SOLUTION:. JSON can use an array of solutions:

[
  {
    "Task": "Your task",
    "Solution": [
      "An acceptable solution",
      "Another acceptable solution"
    ],
    "Errors": "Possible errors and their potential severity"
  }
]

For direct agent imports, the legacy JSON keys remain presentation for Task and recommendations for Solution. Both must be nonempty strings.

Rubrics

Open Rubrics

A rubric turns your Solution and supplied Errors into Decisions and Options. Open the rubric editor from the Output card to review and edit the draft without leaving NOHARMKit.

You supply: Imported items and any existing criteria or ratings you want the agent to use.
Expected output: A rubric version with Decisions, appropriate and inappropriate Options, and validation results.
Before continuing: Use Open Rubric Editor on the Output card. Check Decisions, Options, appropriateness, and harm levels; save, then close the editor. Choose the saved version before autograding.

Prompt for my-benchmark:

Use noharm-rubrics for project "my-benchmark". Draft rubrics for my imported items and help me review them.

Responses

Open Responses

You supply: Use the checklist and Copy model selection, or give your agent the full model list. OpenRouter supports testing models from multiple organizations.
Expected output: One stored response for each expected case, model, and declared trial.
Before continuing: Check missing and failed responses. Decide whether to retry them before comparing models.

Prompt for my-benchmark:

Use noharm-responses for project "my-benchmark". Collect responses using the agreed models and declared trials.

Autograding

Open Autograding

The judge compares each stored model response with your chosen rubric and maps it to Decisions and Options. Confirm the rubric version and judge model, then inspect the mappings and failures.

You supply: Stored model responses, the rubric version to use, and your chosen judge model.
Expected output: Mappings from each model response to rubric Options, with quoted evidence.
Before continuing: Inspect a complete and an incomplete model response. Investigate failed or out-of-date mappings before scoring.

Prompt for my-benchmark:

Use noharm-autograding for project "my-benchmark". Grade the stored responses; confirm the rubric version and judge model with me.

Scores

Open Scores

Scores are calculated deterministically from the stored judgments. Review Recall, Precision, F1, harm, and coverage as separate measures.

You supply: The stored judgments for the intended rubric version. This step needs no model call.
Expected output: Recall, Precision, F1, and harm for each scored model response, with coverage information.
Before continuing: Trace unexpected scores back to the model response and mapping. Missing model responses are not zero scores.

Prompt for my-benchmark:

Use noharm-scores for project "my-benchmark". Compute scores and check coverage.

Leaderboard

Open Leaderboard

The leaderboard ranks models by Mean F1 across their available scored responses, including partial and unverified results. Compare adequacy and harm alongside coverage and review status.

You supply: Scores already stored in this project. There is no separate generation step.
Expected output: A model comparison across the project's items.
Before continuing: Check coverage, harm, and review status alongside F1. A draft comparison is not a clinically validated result.

Results and review

Adequacy and harm describe different parts of model performance. Read them alongside coverage and review status to understand what the results support.

Adequacy

Recall, Precision, and F1

Safety

Harm, reported separately

Evidence

Coverage and review status

After a rubric edit

Save reviewed copy
Prepare fresh project/run
Autograde intended rubric
Recompute scores

LLM draft means model-produced. Validation is not clinical review.
Provisional means the evidence has limits; an average does not remove them.
Missing or failed responses are not zero scores. Check coverage before comparing models.
Saving a reviewed rubric does not update existing judgments. Preserve earlier outputs and use a fresh project/run for rejudging. Verified resume for an interrupted direct OpenAI run can fill missing jobs only; it does not replace failed judgments or accept changed rubrics.
A completed pipeline run and a completed review are separate.