Optional local execution · keep collection separate from acceptance

Run a worksheet. Keep the failures.

Export prompts once, collect responses from your chosen API or local model, then return to the Test Bench for matched import and human review. No account or inference service is hosted here.

Dry-run is the default. The downloaded program makes no request or file write unless you add --run. With that flag, prompts go directly from your computer to the endpoint you configure, under its privacy policy and costs—not through PromptKB. Inspect the code and worksheet before running. A chat subscription does not necessarily include paid API access.

1. Export and inspect

1. In the Test Bench, prepare your cases, both prompts and checks. Export a response worksheet from Import a response batch. Use a separate model conversation per prompt; this runner sends one user message per request without previous-row context.
2. Install Node.js 20 or newer. Save runner.mjs and its license locally. No npm packages, browser API key entry, shell execution from worksheet content or remote code loading are used by the runner.
3. Choose an endpoint/model that you are authorized to use. It must support a non-streaming OpenAI-style POST /chat/completions, messages, max_tokens and a single text choice. This is not a universal provider adapter: Responses/Anthropic APIs, tools, images, streaming and models requiring different token parameters are not supported.

Start with one row and a small output limit. The model may reject optional temperature or seed parameters; omit unsupported options. Provider defaults otherwise apply. Different local chat templates, quantization, model versions and defaults can change responses.

2. Dry-run before sending

node runner.mjs --input worksheet.json --output completed.json \
  --base-url https://provider.example/v1 --model YOUR_MODEL \
  --limit 1 --max-output-tokens 256

The summary shows selected row indices, prompt character count and maximum requested output tokens. It does not print prompts or case names. Input tokens and prices are unavailable before execution; characters are not tokens. This is not a hard spending cap. Set limits at your provider as well, including any input/reasoning charges.

Only each row's rendered prompt is sent—not its expected checks, other responses, manual criteria or project decisions. If you placed answers inside your prompt yourself, they will still be sent. Existing nonblank outputs are skipped. --side A, --side B or --side both selects variants; --limit bounds new calls in this invocation.

3. Execute explicitly

If your endpoint requires a key, supply it through PROMPTKB_API_KEY, never command arguments, endpoint query strings or worksheet content. For example, in Bash:

read -rs -p 'API key: ' PROMPTKB_API_KEY; echo
export PROMPTKB_API_KEY
node runner.mjs --input worksheet.json --output completed.json \
  --base-url https://provider.example/v1 --model YOUR_MODEL \
  --limit 1 --max-output-tokens 256 --run
unset PROMPTKB_API_KEY

A loopback model server can use --base-url http://127.0.0.1:PORT/v1 without a key. Other endpoints require HTTPS. URLs with credentials, queries or fragments and HTTP redirects are refused. Environment variables are not a secret manager; processes with sufficient permissions may read them. The runner does not intentionally save the credential, request headers or raw provider error bodies. Inputs and successful model outputs remain unredacted and may themselves contain sensitive text.

Requests are sequential. Default timeout is 60 seconds, configurable from 1–300. Maximum requested output is 1–4,096 tokens (default 256); successful text must fit the Test Bench's 20,000-character field. Response bodies are capped at 256 KB. Unsupported/empty responses, HTTP failures, timeouts and network errors stop the invocation. A length-limited output is preserved and labeled in the receipt—not repaired, stripped of fences or counted as accepted.

4. Resume without silently repeating calls

node runner.mjs --input worksheet.json --output completed.json \
  --base-url https://provider.example/v1 --model YOUR_MODEL \
  --max-output-tokens 256 --resume --run

Use the original input and the existing output. Resume requires the same assignments and request settings, though side/limit may change. The runner never overwrites an unrelated existing output. It uses a per-output lock and atomic checkpoint replacement with restrictive permissions; it writes an in-flight marker before sending a request. Completed rows are never repeated automatically.

Uncertain does not mean uncharged. A timeout, dropped connection, parse failure or interrupted process may follow successful provider processing. Resume stops on unresolved attempts. After inspecting them, --resume --skip-uncertain --run explicitly leaves those rows blank and continues untouched rows; it does not retry them. A deliberate later rerun may incur duplicate charges. Keep the old receipt and prepare a separate worksheet for that decision.

If the process was killed, a .lock file may remain. Confirm no runner is still using that output before removing only the lock. Do not remove in-flight attempt records to make a request look unattempted. File checkpoints cannot form a transaction with a remote provider, guarantee survival of every power loss, or verify provider billing.

5. Import, review, decide

1. Open the completed checkpoint through the Test Bench's response worksheet importer. It remains worksheet version 1 or 2; runner receipts are extra top-level metadata. Preview validates assignments against your current project before confirmation.
2. Blank/unresolved/skipped rows are not imported or counted as successes. Recorded responses need new manual review. Raw fences, wrong types and factual mistakes remain visible.
3. Inspect source grounding as well as automatic checks. Save a scoped project recommendation and export a handoff.
4. Keep the completed worksheet separately. Project import retains response text and the requested model label, but does not store the runner's timings, settings or token receipts. Those receipts are unsigned local metadata, not verified model identity, execution or approval inside the project.

Actual exercise: six calls, not six successes

We ran the existing three fictional support messages with both prompt variants using Qwen2.5-0.5B-Instruct Q4_K_M through CPU-only llama.cpp b12081. First two requests were checkpointed; resume collected the other four. A final resume after stopping the local server made zero requests and left the checkpoint unchanged. The temporary server was loopback-only and resource-limited; it is now stopped, not a public inference endpoint.

These outputs were actually generated by that local model, not authored as expected answers. The cases, contract and subsequent review were agent-authored; this is still not a blinded review, representative dataset, held-out evaluation, statistical benchmark or cloud-provider compatibility test. One request per prompt, temperature 0, seed 42, maximum requested output 160 tokens and context 1,024. No paid model API was used; existing host/subscription operating costs remain.

Hold, not deploy. A passes 0/3 exact checks and is accepted on 0/3; B passes and is accepted on 1/3. B formats raw JSON consistently here but still drops explicit values and guesses an ambiguous order. Formatting improvement is not sufficient source grounding. Both variants have three current recorded cases; all three pairs are reviewable.

Local provider-reported usage: 867 prompt + 335 completion = 1202 total tokens. Summed observed request duration: 43.058 seconds. These are six request receipts, not controller-agent tokens, billed costs or a general speed benchmark. All six reported finish_reason stop. Model-service peak cgroup memory: 576,987,136 bytes, including cache—not process RSS.

```json
{
  "order_id": "ORD-1042",
  "issue": "cracked mug",
  "refund_requested": "yes",
  "amount": "29.90",
  "currency": "EUR"
}
```

{
  "order_id": "ORD-1042",
  "issue": "damaged_item",
  "refund_requested": true,
  "amount": null,
  "currency": null
}

```json
{
  "order_id": "123456789",
  "issue": "late",
  "refund_requested": "yes",
  "amount": "100.00",
  "currency": "USD"
}
```

{
  "order_id": null,
  "issue": "late_delivery",
  "refund_requested": false,
  "amount": null,
  "currency": null
}

```json
{
  "order_id": "ORD-201",
  "issue": "charged twice for $49",
  "refund_requested": "do not know",
  "amount": "$49",
  "currency": "USD"
}
```

This JSON object represents the customer message in a structured format, including the order ID, the reason for the charge (twice charged for $49), the refund request, the amount charged, and the currency.

{
  "order_id": "ORD-201",
  "issue": "duplicate_charge",
  "refund_requested": false,
  "amount": null,
  "currency": null
}

Scoped decision and next evidence

Do not select either tested variant for unattended extraction with this local configuration. Three paired cases are current: A passes 0/3 exact checks and is accepted on 0/3; B passes and is accepted on 1/3. B produces raw JSON consistently here, but loses an explicit amount/currency and guesses an ambiguous order. Those source-grounding failures matter more than formatting. These are six actual single local generations, not statistical repeat sampling, a representative evaluation or proof about other models/providers.

Before using any configuration on real customer records, test a broader permitted source set with unseen ambiguity/missing-value cases and independently reviewed expectations; repeat generations and compare consistent settings. Fix or replace the tested configuration if it still guesses targets or drops explicit values. Use a provider you are authorized to send the data to, set provider-side spending limits, and inspect uncertain requests before any deliberate rerun.

The reproducible example's recording/review date is an artifact date, not each model request's completion time. Exact per-request device timings are in the original completed worksheet. The project stores the requested model label; the execution file identifies the locally loaded weights/server.

Next: three repetitions on four new cases show why identical outputs can still be wrong. The Test Bench now inspects earlier/current-prompt recordings together without changing the latest-only matrix. Keep this original six-call example separate; the later exercise uses different cases and sampling settings.

Take the complete evidence

• Unrun project and empty worksheet
• Actual six-response checkpoint with settings/timing/token receipts
• Model/server sources, hashes, license references and resource observations
• Reviewed project with a hold recommendation
• Full comparison report and single-file offline handoff
• Hosted reading view

The runner is MIT-licensed; its license does not grant rights to send third-party data to a provider or redistribute third-party model weights. Model/server artifacts are linked, not redistributed here. Task examples retain their existing free-reuse permission. Review provider terms and source permissions before processing real customer records.

Manual batch workflow · Privacy and storage · Return to the Test Bench