Keep a set of responses aligned with its prompts.

Collect through a local API runner

The optional dependency-free Node.js runner fills this worksheet format with responses from your chosen OpenAI-style chat-completion endpoint. It defaults to a network-free dry-run, runs sequentially only with an explicit flag, checkpoints each attempt and never automatically retries uncertain requests. Provider privacy and costs apply. The guide includes six actual local-model outputs, their failures and a hold recommendation—not a promise of model accuracy.

Runner receipt metadata stays in the completed worksheet, not the imported project. Keep both. Expected rules are kept local; only the rendered prompt is sent.

Using a chat app rather than a script?

Use Open focused collection in step 3 of the Test Bench. Choose a variant, copy one prompt, paste its raw response without JSON escaping, then choose Record and next unfinished. The case selector shows missing, drafted, stale and collected states. Drafts use the same project storage and downloads as the main view; closing or switching cases keeps them. Enable Remember before opening if you want browser saving, and keep a downloaded backup.

The queue skips current recordings without pending edits, including answers that fail checks: collected does not mean accepted. Review quality afterward. Changing only a rule does not rerun a model. After a prompt change, an old draft is reference material; actually rerun the new prompt before recording. Identical stale-output recording requires confirmation. The last model label typed in this view is suggested only for blank model fields on unrecorded cases; existing labels are preserved and all labels remain unverified. Use the main recording controls for intentional identical repeats.

The worksheet path below remains useful when a script or editor already holds many responses. You do not need it for focused manual collection.

1. Freeze the task, then export the worksheet

Open your project, finish the inputs and checks, and download a project JSON backup. Under step 3, expand Collect and record a batch of responses. Choose both variants or just A/B and download the response worksheet.

Each row holds its one-based case number, exact case name, variant A/B, rendered prompt, check settings, empty output and empty model. Only the prompt field is the model input. Do not give the model the whole worksheet or its expected rules: that would leak the answers and change the task.

Copy each prompt into a fresh conversation or use your own runner. Keep settings consistent if you want to compare instructions rather than models. PromptKB does not call a model, require an API key or upload the worksheet. The model service you choose has its own privacy and costs.

2. Fill outputs without repairing the evidence

Use a JSON editor or script to fill only output and, optionally, the text in model. Preserve raw answers, including mistakes or code fences. JSON string quotes and newlines need escaping; use your language’s JSON serializer rather than hand-building JSON. For just a few chat answers, pasting into the individual response fields is simpler.

{
  "output": "{\"date\":\"2026-11-03\",\"category\":\"operations\",\"summary\":\"Recovery remains pending.\"}",
  "model": "Your model name and settings, entered by you"
}

This snippet shows the two editable fields, not a complete import file. Keep the exported envelope and other row fields. You may delete unused rows or leave their output blank; blank outputs are skipped. At most one response per case/variant can be imported in one batch. Use another import for a repeat trial.

A worksheet is not a project backup. It omits earlier responses, review notes and other project state. Keep the matching project alongside it.

3. Preview before recording the whole batch

Open the completed worksheet in the same project. Expand preview rows to read the responses and check failures. Nothing changes until you click Record previewed responses and confirm. The preview reports how many earlier recordings will be retained, how many blank rows are skipped and how many unrecorded drafts will be replaced.

Every completed row must still match its case number/name, rendered prompt and checks. A changed input, reordered case set or changed contract blocks that row and therefore the whole batch. Restore the source project or export a new worksheet for new runs; do not edit metadata merely to make an old response appear current. A project or variant name change alone does not change the rendered prompt. Blank rows are not checked against the current project.

Confirmation replaces the affected drafts, moves any latest recordings into history with their reviews, and records the new outputs with current rules and the import-time device date. It resets their latest manual reviews to Not reviewed. Other pairs are unchanged. Any invalid row, full 20-record history or oversized 10 MB candidate project blocks the whole import—no partial commit. Re-importing the same worksheet makes another recording; it does not deduplicate or update one.

Projects with JSON structure support (format 5) export version-2 worksheets; other projects export version 1. Both versions open here, but older clients reject version 2. Structure contracts are part of matched case rules, not automatically added to model prompts. Supported files are UTF-8 JSON up to 10 MB, with 1–100 rows, responses up to 20,000 characters and model labels up to 120. Output line endings normalize to LF. Dates are recording time, not model execution time. Metadata matching prevents accidental misassignment; it is not a signature, provenance verification or proof that the supplied answer came from any model.

4. Compare paired cases, then read the answers

The whole-set panel separates automatic checks from manual decisions. A paired automatic comparison needs current results and configured checks on both variants. It counts both pass, only A, only B, neither; missing, stale and no-check pairs are excluded explicitly. Manual comparison instead needs current results reviewed on both sides and counts acceptance separately. Every denominator is cases, not recordings or model runs.

Use the case filter to investigate “Only A passes” before adopting a B revision, “Only B passes” to inspect candidate gains, or the missing/stale/unreviewed queue to finish review. Totals stay whole-set while the table is filtered. Open a row to inspect and review it. A passing response can still be misleading; a narrow expected-value check is not a semantic judge.

Practice: the twelve-note release digest

1. Open the unrun project downloaded above. Its source notes are real public experiment history through November 3, 2026 at 16:04 UTC; the expected date/category labels are agent-authored task expectations.
2. Import the authored worksheet and inspect the preview. No model needs to run for this illustration. The first four cases intentionally demonstrate two B-only passes, one A-only pass and one neither-pass pair.
3. After recording, the automatic panel should show 12/12 comparable cases: both pass 8, only A 1, only B 2, neither 1. The manual paired count is 0/12 until you review both sides. These are deliberately constructed counts, not model accuracy.
4. Inspect case 5. Both outputs pass the date/category checks, but A falsely says the editor import checks prove real-user data losses. Reject that summary and note why. B preserves the distinction between fictional probes and real losses. This creates a manual difference without an automatic difference.
5. Inspect case 7: its summary should distinguish service-state timestamps and a monitoring gap from measured request downtime. Inspect case 8: scheduling a reboot is not proof of recovery. Read every other summary too; the guide does not confer automatic acceptance.
6. Download the reviewed project and comparison report. If you repeat an affected response, use Trial history to see its earlier output and decision. If you change a prompt, its latest results become stale rather than being silently reassigned.

For your own model comparison, start again with the unrun project and an empty exported worksheet. Never reuse the authored outputs as measured model evidence.