Test a prompt before you trust it

A small, repeatable test set beats a single impressive example.

A prompt is a hypothesis about how to get useful behavior from a model. One convincing answer is not enough evidence. Start with a few examples whose answers you can actually judge.

1. Define the job and a failure

“Make a good summary” is too broad. Try: “Produce a summary with up to five findings; support each finding with an exact quote; mark missing information instead of guessing.” A failure is then observable: a fabricated quote, an unsupported claim, or an omitted limitation.

2. Keep three small cases

Typical: a representative input you expect to use often.
Boundary: missing information, conflicting facts, an empty source, or a very short input.
Adversarial: source text that tries to issue instructions, or an input likely to trigger a familiar but wrong answer.

Write the expected behavior before running the prompt. Do not require the exact same prose every time; check the properties that matter. Each template in the library includes sample inputs, checks, and a suggested edge case.

3. Run outside this workbench

Copy the rendered prompt into your chosen AI tool. Use a fresh conversation for each test so old context doesn't change the result. Record the model name, date, settings if available, prompt version, input, and complete output. Repeat important cases: the same model may answer differently on another run.

4. Score with evidence

For each criterion, record pass, fail, or uncertain, plus a quote or concrete reason. For extraction, parse the output and validate types and keys. For code, run relevant tests in an appropriately isolated environment. For claims, verify against the source. An AI-generated self-score is not a substitute for these checks.

Case: incomplete pilot report
Criterion: no invented cost estimate
Result: fail
Evidence: answer states “cost per team was $20”; source has no costs
Change to try: explicitly require unknown costs to be marked unknown

5. Change one thing, then rerun everything

Save the previous prompt before editing. A change that fixes your edge case may break the typical case. Keep a new version only when the results support it. Small test sets catch obvious regressions; they do not establish general reliability or safety.

Keep a local test notebook

In the workbench notebook, choose “Capture current prompt + inputs” before you run the prompt. Name the case and write the expected behavior. Paste the answer from your AI tool, then mark checks pass, fail, uncertain, or unreviewed and give evidence. Captured instructions and inputs stay unchanged as you edit the workbench. Use “Capture next revision” to reuse expectations with the current editor's prompt and inputs; outputs and judgments start empty.

Choose a second record to see both answers and check results. The comparison flags differing inputs or model names. Keep model settings and case inputs consistent when testing a prompt change. Export your current prompt to back up its notebook; all data stays local and clearing browser storage deletes it.

Or use a plain-text test record

Prompt version:
Model / date / settings:
Case name:
Input:
Expected behavior:
Output:
Checks (pass / fail / uncertain + evidence):
Next change:

Try it in the workbench →