{ "format": "promptkb-test-bench", "version": 4, "rubric": [ "Grounding: every capability or outcome claim is supported by the source; test or notification receipts are not presented as measured human or model benefits.", "Reader value: explains a concrete workflow or user problem addressed, not only implementation activity.", "Next action: gives an explicit next step with a named control or destination a reader can find, rather than merely listing capabilities.", "Limits: states a material limitation relevant to the change without overstating guarantees." ], "name": "Product release briefs — 3 November 2026, agent editorial review", "variants": [ { "name": "A · concise update summary", "template": "Write a concise product-update summary for existing PromptKB users from this source note. Keep it under 110 words. Preserve factual limits.\n\nSource note:\n{{input}}" }, { "name": "B · action-oriented brief", "template": "Write a product-update brief for existing PromptKB users from this source note, under 110 words. Use Change:, Why it matters:, Next:, and Limit:. Give an explicit next action with a named control or destination. Preserve material evidence limits. Do not claim measured human benefits, model superiority or audience reach from authored exercises, local tests or notification receipts.\n\nSource note:\n{{input}}" } ], "cases": [ { "required": "", "forbidden": "", "maxWords": 110, "jsonCheck": { "enabled": false, "expected": "", "exact": false }, "name": "Test Bench", "input": "Public product update 2026-11-03T15:00:48Z\nPrompt Test Bench turns isolated prompts into a reviewable comparison project\nPublished /lab/ with two instruction variants, up to ten reusable cases, literal required/forbidden phrases and word caps, copied prompts, pasted response drafts, explicit prompt-bound recording, manual reviews, stale-result detection and JSON/Markdown exports. It is local-only, makes no model calls, sends no interaction events and is linked from the workbench; privacy and sitemap are updated. The agent worked through public reboot facts and a fictional missing-evidence case with six clearly labeled agent-authored responses. The task now retains the case set, exact prompt/response pairs and review rationale together instead of six manually managed outputs. Navigation was improved after the first exercise. Nine grouped local/live workflow checks passed, including 320/390/1280px axe/layout and no content-bearing requests; six changed live assets matched source, and 102 Python/66 Node regressions passed. These are tool/dogfood results, not statistical prompt superiority, verified model runs, human adoption or revenue. The independent 15:00 snapshot was standard-verified and includes all four lab assets; no additional application/browser/OS restore drill was performed.\nPublic product destination: https://promptkb.com/lab/", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 110, "jsonCheck": { "enabled": false, "expected": "", "exact": false }, "name": "Typed extraction checks", "input": "Public product update 2026-11-03T15:31:57Z\nJSON extraction checks catch unusable records and explain the failing fields\nPublished typed expected-JSON checks, exact/subset object matching, per-field differences and a reusable three-case support-extraction starter with a practical /lab/extraction/ walkthrough and download. Reports now retain current case rules as well as results. A baseline non-JSON response passed all three literal checks; the new six-response authored exercise catches type errors, invented missing values and ambiguity while allowing reordered object keys. This is demanding tool use with fictional inputs/authored outputs, not independently sampled model performance. The local and live extraction workflows passed, including prompt copying without expected-answer leakage, reviews, exports, rule iteration, stale snapshots, reopening, version-1 migration and focused accessibility/narrow-layout checks. The existing incident workflow also passed locally. Visual review prompted aligned response editors. Deployment passed 102 Python and 70 Node checks; eight public assets match source, publication is healthy, and no units failed. No controller/backend/collection change or additional recovery drill was made. Current hourly backup predates this release; browser-local projects remain outside server backups. A post-release agent narrative-edit error omitted a JSON key: the 15:32:34 collector marked the narrative stale and retained the prior version while continuing measurements. The source was corrected and JSON-validated before 15:34:13; the automatic 15:34:36 snapshot restored healthy publication. This was a narrative publication interruption, not observed product downtime.\nPublic product destination: https://promptkb.com/lab/extraction/", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 110, "jsonCheck": { "enabled": false, "expected": "", "exact": false }, "name": "Spreadsheet case sets", "input": "Public product update 2026-11-03T16:37:56Z\nSpreadsheet case sets become reviewable local imports, not a series of manual form entries\nPublished CSV/TSV file and paste intake with column mapping, full input/rule previews, optional independent rule copies and confirmed append/replace. Existing recordings survive append, matching names are not merged, and replacement preserves variants while explicitly removing cases/history. New cases remain unrun. Capacity grows from ten to fifty cases without new format-3 fields; older releases reject larger projects. The standalone /lab/datasets/ guide supplies twelve already-public notes as a CSV and ready digest project, with authored date/category labels and manually reviewed summaries rather than model benchmarks. Direct baseline validation rejected that twelve-case project under the old limit. Five local/live workflow groups passed: actual multiline CSV, recording preservation, cancel/re-preview, replace/reopen, TSV inheritance and focused layout/accessibility/privacy checks. Earlier history workflow also passed locally. Deployment passed 102 Python/82 Node checks; twelve live assets match source, publication is healthy with unchanged source-transition count, and no units failed. No controller/backend/collection change, external submission or additional restore drill occurred; latest scheduled backup predates this release.\nPublic product destination: https://promptkb.com/lab/datasets/", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 110, "jsonCheck": { "enabled": false, "expected": "", "exact": false }, "name": "Support extraction pack", "input": "Public product update 2026-11-03T17:40:00Z\nA reusable sixteen-case support pack ships, followed by one acknowledged search notification\nPublished a script-free support-extraction regression resource with fictional cases, two prompts, typed expectations, coverage/rationales and portable unrun project, generic JSON, CSV and rendered-prompt downloads. Free reuse permission applies to the pack, not the whole site. Policies for refund intent, issue priority and localized amounts are explicit and adjustable rather than universal truth. Local/live exercises retained an old rejection when the same authored amount response became acceptable under a revised contract, and recorded a deliberately authored injection-following failure without claiming measured attack resistance. Four browser workflow groups and 105 Python/92 Node tests passed; nine public product assets match source, verification file was checked separately, publication remains healthy and no units failed. After reviewing current IndexNow documentation/FAQ/terms and preflighting owned canonical pages, one POST at 17:38:07 received HTTP 202 at 17:38:09 for /packs/support-extraction/, /lab/ and /lab/batch/. This means received with key validation pending, not indexing/ranking/visitors. No retries, other endpoint or community posts. The redacted receipt and authoritative duplicate-guard ledger are retained with source so later workspace backups preserve the commitment; the latest 17:00:56 backup predates this release. No controller/backend/timer/collection changes or new restore drill.\nPublic product destination: https://promptkb.com/packs/support-extraction/", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 110, "jsonCheck": { "enabled": false, "expected": "", "exact": false }, "name": "Saved-baseline revision review", "input": "Public product update 2026-11-03T18:43:10Z\nSaved-baseline comparison exposes whole-set regressions and a dangerous automatic pass\nPublished read-only previous-project comparison below the Test Bench matrix, with one variant selected from each project, unique exact case matching, current-candidate-rule checks, explicit exclusions and full-set report export. Added a script-free customer-reply walkthrough with authored baseline/candidate and unrun downloads. The first candidate has 4/7 comparable cases: one gain, one regression and two both-pass. One both-pass reply requests unnecessary payment-card data and has a saved rejection; the new saved accept-to-reject queue exposes it without inventing a common-rubric quality score. Editing the shared prompt then collecting seven authored repairs retains earlier reviews and produces 5/7 comparable cases, one gain and four both-pass; changed/new inputs remain unpaired. These are constructed workflow results, not independent model samples. Visual review aligned raw outputs ahead of variable-length metadata. Four local/live browser groups passed, including saved-copy isolation, report completeness, review navigation, stale evidence, drafts, invalid/delayed loads, reload/replacement, narrow layout/axe and no content-bearing requests. Deployment passed 105 Python/100 Node tests; ten public assets match source, publication is healthy with four unchanged source transitions and no units failed. Verification file and single prior discovery attempt remain preserved; no external submission, controller/backend/timer/data-collection change or additional restore drill. The 18:00 standard-verified backup predates this release.\nPublic product destination: https://promptkb.com/lab/revisions/", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 110, "jsonCheck": { "enabled": false, "expected": "", "exact": false }, "name": "Context-bound manual criteria", "input": "Public product update 2026-11-03T19:17:20Z\nManual criteria keep the basis for a judgment when review policy changes\nPublished shared criteria, explicit per-criterion assessments, current-context eligibility and retained review rounds in the local Test Bench, plus a script-free four-reply guide and unrun practice project. Sixteen authored initial assessments yield two manual acceptances versus three automatic passes; the payment-data request is a deliberate semantic rejection among literal passes. Tightening criteria and then a case's word cap invalidates old judgments without changing recorded prompts; canceled restart keeps everything, while confirmed re-review preserves old criteria/ratings/checks/decision/note. The exported later-stage example has one current acceptance and three out-of-date reviews, not the earlier two-acceptance state. New responses retain these contexts in Trial history. Format 4 imports older projects without invented assessments; a direct old-validator probe rejects the new version. Four local/live workflow groups passed, including storage/reload, staging, draft separation, review lifecycle, exports, baseline/history, narrow layout/axe and content-free requests. Existing baseline and extraction browser workflows also passed locally; one extraction evidence caption still said v3 while the assertion checked v4, and the caption was corrected afterward without rewriting the evidence. Deployment passed 105 Python/107 Node tests. Fifteen assets match live source, publication is healthy with four unchanged transitions and no units failed. The single earlier discovery attempt and verification file remain preserved. No controller/backend/timer/dependency/data-collection change, new model call, external submission or restore drill. Latest 19:00 verified backup predates this release.\nPublic product destination: https://promptkb.com/lab/review/", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] } ] }