Prompt Test Bench · local-first
Find out what your prompt misses.
One impressive answer is not a test. Compare two instructions against the same cases, keep the responses beside their exact prompts, and decide what is worth reusing.
Start with a reusable task: meeting notes → action register or support tickets → typed records. Preview a full unrun pack below without downloading and re-importing files.
- Define two variants and representative cases.
- Run each prompt in your chosen model.
- Review the whole set, then export the comparison.
Start from a reusable task pack
Owned public examples, fetched only when you preview. No private project content is read by the server. Read the task policies and reuse terms.
Preview changes nothing. Confirmed loading replaces this tab's project, including unsaved work, but turns Remember off and leaves the previous browser-saved project intact. Download your current project first if you need it. To save the starter later, explicitly turn Remember on; otherwise reload can restore the older saved project. Packs contain no model responses or scores.
Your comparison project
Saving is off by default. JSON includes prompts, cases, drafts and recorded responses—not redacted. Keep a downloaded copy of important work. This project format is separate from Workbench template files. Opening JSON normalizes CRLF/CR line endings to LF for editing. Up to 50 cases, 20 recordings per case/variant and 10 MB project files. Browser saving may fill up sooner; keep downloaded backups. Projects without structure contracts or saved project decisions remain version 4. Adding structure upgrades to at least version 5; saving a project decision upgrades to version 6. Versions 5–6 use version-2 response worksheets. Version 6 retains exact decision-evidence snapshots, which can contain sources/cases later deleted from the current set. Older clients reject these versions rather than silently dropping checks. Versions 1–4 still open without invented structure contracts; earlier reviews remain in their original context. Removing a contract or decision does not downgrade its project version. Keep original files before migrating.
1. Change the instructions, not the test
Use {{input}} wherever the case input belongs. Other braces are literal text. Recorded responses become stale if their rendered prompt changes; they are kept, not silently reassigned.
2. Build a useful test set
Import a case dataset from CSV or a spreadsheet
Read locally, map columns, preview every case, then append or replace. Walkthrough and a 12-case public-history dataset.
Comma-separated files support quoted commas, doubled quotes and multiline cells. For spreadsheet clipboard cells, choose Tab. Line endings normalize to LF. Up to 50 data rows and 30 columns. Empty lines are skipped; inconsistent row lengths are rejected, not partially imported. This staging text is not saved in the project.
Off means no phrase checks or word cap. Copies are independent, not shared rules. A mapped expected-JSON column replaces expected values and exact/subset matching, retaining a copied structure contract and requires JSON for every imported row; a blank cell skips expected-value comparison, not an expected empty string; syntax and any inherited structure still apply. Choose Not mapped to avoid this.
Nothing changes until you confirm import. No matching or merging by name. New cases start without drafts, results or history. Both prompt variants and project name stay unchanged. Download project JSON for a complete backup; a CSV dataset is not a project backup.
One literal phrase per line. Checks ignore letter case and match substrings; words are separated by whitespace. They do not verify truth, meaning or instruction-following. Changing these rules rechecks recorded responses.
For extraction, classification and other structured work. Try the support-ticket walkthrough and downloadable starter.
Require fields, types and nonempty text without fixing one exact answer. Uses an explicit JSON Schema-style subset, not a full JSON Schema validator. Unknown keywords are errors; no remote references or code execution. Blank means no structure check. Changes recheck recordings and make bound reviews out of date, not stale prompts. Checks are not inserted into model prompts. Supported keywords and action-register example.
After confirmation, copies only this structure and enables JSON checks for all cases when nonblank. Existing expected values, phrase checks, word limits, recordings and review history stay. Blank removes structure contracts, without disabling other JSON checks.
Expected JSON field: blank skips expected-value comparison; JSON syntax and any active structure contract are still checked. Otherwise compare these values and types: objects ignore key order; arrays require the same order and length. Without exact match, extra object keys are allowed at any depth. Strings are case-sensitive; null differs from a missing key. No code fences or commentary. This is an expected-value check, not JSON Schema or proof of correctness. Uses JavaScript JSON parsing/numbers; duplicate keys keep the last value. Up to 12 differences shown.
Define shared manual review criteria
Use explicit criteria for meaning and task usefulness that text/JSON checks cannot judge. Try the customer-reply review walkthrough. These criteria apply to all cases and both variants; they are not inserted into prompts or sent to a model.
Blank means no criteria. This editor is a draft until Apply; only applied criteria enter saving/exports. Changing criteria, their order, or a case’s automatic checks makes bound reviews out of date, without changing prompts or deleting prior judgments. Start a review below to bind assessments to current criteria/checks. Earlier versions have no criteria snapshot: their simple decisions remain usable only while no criteria are configured.
3. Run both prompts and judge the responses
Copy a prompt, run it in a fresh chat with the same model/settings, then paste the response. Model labels are your notes, not verified execution metadata. For manual criteria, start a review, assess every criterion, then explicitly accept or request revision. All criteria must meet your judgment before acceptance is available; passing assessments never auto-accept. Editing an assessment resets the overall decision. Record the result to bind it to the current prompt and rules. Recording again keeps the earlier response and review in Trial history. No model is called here.
Collect and record a batch of responses
Keep responses aligned with the exact case, prompt and checks you exported. Worksheet guide and a completed, explicitly authored example. No API or model is called.
Run each row’s prompt separately. Fill only output and optional model, leaving identity/prompt/rules unchanged; the rules are for checking, not for sending to the model. Blank outputs are skipped. You can remove unused rows. JSON is useful for scripts and editors; for a few manual chat responses, the individual fields above avoid JSON escaping. Keep your project JSON alongside the worksheet.
Prefer API collection? The optional local worksheet runner sends each prompt directly to your chosen endpoint only with an explicit run flag. Dry-run first, bound requests and keep checkpoints; this browser makes no model call. Provider privacy and costs apply. Download runner.
Preview does not change your project. Confirmation records all completed rows together, replaces their response drafts and resets their latest reviews to Not reviewed. Previous recordings and reviews stay in history; untouched pairs stay unchanged. A changed case name/order, prompt or checks—including a structure contract—blocks affected completed rows. Version-5/6 projects export version-2 worksheets so older clients cannot silently drop structure checks. Any full history or oversized project blocks the whole batch. Re-importing creates another recording, not an update. File text and preview are not saved. Matching metadata does not prove that a model ran the prompt.
4. Compare the whole set
Only the latest recorded response per case/variant counts, and only if its prompt is current. Earlier trials are excluded from these totals. “All checks pass” means every configured text/JSON check passed, not that the answer is good. A case with no checks has no automatic pass. Manual acceptance is separate; not recorded and stale cases stay visible. With criteria, only complete current assessments with an explicit overall decision qualify. Changed criteria or case checks exclude old bound judgments until re-reviewed; saved decisions remain visible in their context.
| Case | Variant A | Variant B |
|---|
Find recurring manual-criteria failures
Which requirements need attention?
See where the same criterion fails across cases, then read the supporting response and note before changing the prompt. Release-brief workflow and worked editorial review.
Each criterion counts its own current pass/fail judgments, even if other criteria or the overall decision are unfinished. These are reviewer assessments, not automatic checks, model accuracy or a winner score. Awaiting means this criterion is unassessed in a current review. Excluded means no response, changed prompt, review not started or changed criteria/checks. Drafts and older trials/reviews do not count. Different variants may have different coverage.
| Applied criterion | Variant A | Variant B |
|---|
Inspect evidence
| Open recorded response / review | Criterion evidence | Overall review context |
|---|
The report includes all criteria and cases, regardless of filters, with underlying recorded outputs and notes. It is not redacted or re-importable. View selections are temporary and are not project data.
Review handoff: one read-only HTML file with a case index, criterion links, source/response pairs and saved judgments. Opens offline without the editor; no scripts or remote assets. Includes private content—not redacted. Response/model drafts and earlier recordings are omitted and flagged; current inputs/prompts can still contain private edits. All saved project decisions include their exact evidence snapshots, which may retain deleted cases or older private source text. Latest recordings retain their earlier review contexts. Keep project JSON for the editable, complete backup. See a worked handoff and sharing guide.
Markdown reports include inputs, exact recorded prompts, pasted responses and notes. Share only after reviewing the contents. A project JSON is the editable backup; a Markdown report is not re-importable.
Compare a revision against a saved baseline
Keep the current project open as your candidate. Load a previously downloaded project here without replacing either one. Compare one variant from each across the whole set. Worked customer-reply example and guide.
Both outputs are checked against current candidate rules, not their historical pass rates. Only latest recordings current in their own project qualify. Exact unique case name + input must match; case order may differ. Renames, changed inputs and duplicates are not guessed. Saved reviews are context, not a new common-rubric evaluation. Model labels are unverified; a change is not controlled prompt evidence.
| Inspect case | Current candidate checks | Evidence changes / exclusions |
|---|
Inspect paired evidence
The baseline is read-only, held in this tab only and excluded from Remember and project exports. Reloading or replacing the current project clears it; keep both original downloads. Current edits update this comparison; drafts and earlier trials do not count. Markdown reports contain both projects’ paired inputs, exact recorded prompts, outputs and notes—not redacted, not re-importable.
Save a project recommendation with its evidence
Case acceptance is not a deployment decision. Record your choice, why it fits an intended use, and what remains unproven. See the real-meeting decision example. Nothing selects a winner or authorizes a release automatically.
These form fields are unsaved drafts until Save; project replacement or reload clears them. Save captures current project/case names, sources, variants, checks, criteria and latest recorded responses with their review contexts. It excludes unrecorded response/model drafts and earlier recordings; snapshot editor fields start from retained latest records. Any change to captured evidence marks the decision historical; exact restoration can make it match again. Draft-only edits and trial-history edits do not change that match. Matching is not proof of quality, completeness or approval. Dates and labels are unverified.
Retained private content: up to five decisions keep independent copies of their evidence, including sources/cases later edited or deleted. Downloads, reports and HTML handoffs include all decision snapshots. Deleting current cases does not redact those snapshots. Download before deleting a decision; the 10 MB project limit and browser quota still apply. Decisions are not edited in place: save a new one to preserve the earlier rationale.
Inspect repeatability across retained recordings
One latest answer can hide earlier failures. Include every retained recording for the current prompt, separated by case, variant and exact model/settings label. Three-repeat local-model exercise and guide.
Rechecks use current case rules for all eligible responses, not historical pass rates. Earlier reviews count only when their criteria/checks context still matches. Identical responses remain separate recordings; “distinct” compares exact text, including whitespace and JSON key order. Mixed outcomes means both pass and fail within one case/variant/label group, not a quality score. Missing recordings are not successes. No pooled pass rate or automatic winner.
Labels and dates are unverified. Same label does not prove same model/settings or a new execution; blank labels mean unknown settings. Selected, correlated or manually duplicated recordings are not independent trials. Deleted records, decision snapshots and drafts are excluded. Saved project decisions and HTML handoffs retain only latest recordings: download this separate report plus project JSON to share repeated evidence.
| Case | Variant A recordings | Variant B recordings |
|---|
Repeated evidence
The Markdown report includes every case/variant and retained recording, including excluded prompts, raw responses and review notes—not just the selection above. It is unredacted and not re-importable. Keep project JSON for a full editable backup and runner worksheets for execution receipts.
5. Trial history: what changed?
For the selected case above, compare any earlier recording with the latest. Prompts, responses, model labels, recording-time rules and the earlier review are retained. Dates come from your device; imported dates and labels are unverified. This is local evidence, not an execution log or statistical benchmark.
Up to 20 recordings per case/variant, including the latest. Nothing is automatically trimmed: at the limit, download a backup and explicitly delete an earlier recording. Deleting a case deletes all its trials. Older imported records have unknown recording dates/rules; these are not reconstructed from current settings. Checks shown for saved rules are recomputed by the current checker, not historical software.
Each recorded response can retain up to 10 earlier review contexts plus its latest editable review. Restarting archives the previous criteria, assessments, case checks, decision and note; editing within a review does not keep every keystroke. At the limit nothing is trimmed: download first, then explicitly delete an earlier review. A new response retains all of the old response’s reviews in Trial history and starts without an assessment. Review dates are device-reported start times, not verified completion or model execution times.
What makes a useful prompt test?
Start with an ordinary case, a difficult case and a case with missing evidence. Decide what must stay true before looking at responses. Use the same model and settings for both variants and record repeated trials separately if you need to assess variability.
The support extraction starter uses three fictional customer messages and explicit expected JSON to expose missing, invented or mistyped fields. It starts without responses. The incident example uses public facts from this experiment plus a fictional customer report. It has no pre-scored responses and is not a model benchmark. Literal checks can catch a missing heading or a forbidden claim, but a response can pass them while still being wrong. Read every result.
Each new recording keeps the previous response, exact prompt, rules and review. Trial history lets you inspect these alongside the latest result, including how an earlier output performs under current rules. This is not a full project-version or automatic model-run system; case deletion, project replacement and explicit history deletion can still remove records. Download important work. There is no cloud sync, account, hosted inference or automatic semantic judge. An optional downloaded local runner can collect response worksheets from your chosen API; it is separate from this browser and its provider costs/privacy apply. Changes are in memory unless Remember succeeds or you download the project. Optional workbench interaction counts do not apply here; ordinary page requests still reach the web server.