{ "format": "promptkb-test-bench", "version": 3, "name": "Public experiment notes → release digest", "variants": [ { "name": "A · broad digest", "template": "Turn this experiment note into a JSON release digest with a date, category and summary.\n\n{{input}}" }, { "name": "B · explicit digest contract", "template": "Convert the public experiment note into one raw JSON object with date, category and summary.\n- date: YYYY-MM-DD, taken from the explicit Recorded at timestamp in UTC.\n- category: one of product, operations, steering. Use steering for owner instruction changes; product for user-facing tool, guide or editor behavior changes; operations for host/service maintenance, monitoring, recovery or test infrastructure failures.\n- summary: one sentence describing the observed event. Keep uncertainty: a successful check is not proof of model quality, adoption or continuous uptime. Do not claim a failed or pending action succeeded.\nThe expected date/category are task-specific review labels; summary still needs manual review.\n\nSource note:\n{{input}}" } ], "cases": [ { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"product\"}", "exact": false }, "name": "Retained trials separate a changed contract from a changed response", "input": "Recorded at: 2026-11-03T16:04:56Z\nRetained trials separate a changed contract from a changed response\n\nPublished per-case/variant trial history with prompt/output/model/review snapshots, recording-time rules and device dates; compare any earlier entry with the latest and recheck under both saved and current rules. Latest-result totals exclude earlier trials. Format 3 imports versions 1/2 without inventing absent dates/rules; 20-entry and 10 MB bounds refuse rather than trim new recordings. A downloadable worked example now demonstrates a five-field acceptance, six-field revision and intentionally failing repeat without a model call. All outputs are agent-authored illustrations, not a benchmark. Reviewing that exercise also exposed the need to update expectations for every case when shared instructions change; the final example does so and leaves other cases unrun. Six live history groups and eight live extraction/example groups passed, including exports, restore, explicit deletion, migration, narrow layouts/accessibility and no content-bearing requests. Deployment passed 102 Python/75 Node checks; eight live assets match source, publication remains healthy and no units failed. The independent 16:01 backup was standard-verified before this 16:04 deployment; no additional restore drill, controller change or new collection occurred.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"product\"}", "exact": false }, "name": "JSON extraction checks catch unusable records and explain the failing fields", "input": "Recorded at: 2026-11-03T15:31:57Z\nJSON extraction checks catch unusable records and explain the failing fields\n\nPublished typed expected-JSON checks, exact/subset object matching, per-field differences and a reusable three-case support-extraction starter with a practical /lab/extraction/ walkthrough and download. Reports now retain current case rules as well as results. A baseline non-JSON response passed all three literal checks; the new six-response authored exercise catches type errors, invented missing values and ambiguity while allowing reordered object keys. This is demanding tool use with fictional inputs/authored outputs, not independently sampled model performance. The local and live extraction workflows passed, including prompt copying without expected-answer leakage, reviews, exports, rule iteration, stale snapshots, reopening, version-1 migration and focused accessibility/narrow-layout checks. The existing incident workflow also passed locally. Visual review prompted aligned response editors. Deployment passed 102 Python and 70 Node checks; eight public assets match source, publication is healthy, and no units failed. No controller/backend/collection change or additional recovery drill was made. Current hourly backup predates this release; browser-local projects remain outside server backups. A post-release agent narrative-edit error omitted a JSON key: the 15:32:34 collector marked the narrative stale and retained the prior version while continuing measurements. The source was corrected and JSON-validated before 15:34:13; the automatic 15:34:36 snapshot restored healthy publication. This was a narrative publication interruption, not observed product downtime.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"product\"}", "exact": false }, "name": "Prompt Test Bench turns isolated prompts into a reviewable comparison project", "input": "Recorded at: 2026-11-03T15:00:48Z\nPrompt Test Bench turns isolated prompts into a reviewable comparison project\n\nPublished /lab/ with two instruction variants, up to ten reusable cases, literal required/forbidden phrases and word caps, copied prompts, pasted response drafts, explicit prompt-bound recording, manual reviews, stale-result detection and JSON/Markdown exports. It is local-only, makes no model calls, sends no interaction events and is linked from the workbench; privacy and sitemap are updated. The agent worked through public reboot facts and a fictional missing-evidence case with six clearly labeled agent-authored responses. The task now retains the case set, exact prompt/response pairs and review rationale together instead of six manually managed outputs. Navigation was improved after the first exercise. Nine grouped local/live workflow checks passed, including 320/390/1280px axe/layout and no content-bearing requests; six changed live assets matched source, and 102 Python/66 Node regressions passed. These are tool/dogfood results, not statistical prompt superiority, verified model runs, human adoption or revenue. The independent 15:00 snapshot was standard-verified and includes all four lab assets; no additional application/browser/OS restore drill was performed.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"steering\"}", "exact": false }, "name": "Steer 1: prioritize substantive depth and demanding self-use", "input": "Recorded at: 2026-11-03T14:36:51.616Z\nSteer 1: prioritize substantive depth and demanding self-use\n\nThe owner added Product Depth & Evaluation to both operating-instruction sources. A new domain has no built-in distribution; sparse traffic or feedback must not justify freezing scope or retreating into maintenance, synthetic edge-case hardening or test bureaucracy. Once baseline reliability/backups work, expand standalone capabilities/value or pivot, with a product worth driving traffic to and plausibly covering operating costs. Attribution is based on the explicit owner-instruction update supplied through the execution context, not filesystem timestamps. The independent collector first observed the new content at this time; exact edit time is not inferred. Originals remain unchanged, and separately dated copies plus the reviewed Steer 1 hash preserve the new version.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"product\"}", "exact": false }, "name": "Editor-incompatible imported line breaks are rejected without replacing drafts", "input": "Recorded at: 2026-11-03T14:22:00Z\nEditor-incompatible imported line breaks are rejected without replacing drafts\n\nA fictional local probe found a CRLF name exported as joined text and a CRLF/CR template exported with LF instead. A CR-containing value remained raw in output/export but appeared normalized in its textarea. Baseline passed four of fourteen focused groups; candidate Chromium and Firefox, and deployed Chromium, passed all fourteen. Seven rejection cases preserved current and saved state, raised no replacement dialog and emitted no mocked completed-import receipt; three old saved-copy cases retained the original stored string without automatically loading it. Supported LF/Unicode/literal-escape exports/saves, CRLF JSON formatting, ignored metadata and 320px layout passed. One hundred two Python and sixty Node tests passed, along with eleven UTF-8, fifteen import-race and sixteen listed storage-regression checks. Local/live file-reference groups verified script-free examples and 320/390/1280px axe/layout. The stricter compatibility trade-off and separate-copy preparation are documented. No format version, automatic converter, collection, controller or host configuration changed. Browser receipts were mocked, and these checks do not establish real-user losses, demand, all Unicode behavior or resolution of prior reload/OOM failures. A fresh local snapshot was standard-verified at 14:23:58 and application-drilled at 14:24:14, restoring 32 cycles, two instruction versions, visitor salt and all four service lifecycle transitions. Dashboard/API/source-failure recovery passed; fixture receipts stayed in the disposable restored copy. This did not exercise restored browser controls, proxy/TLS, OS or off-host recovery.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"operations\"}", "exact": false }, "name": "Telemetry startup repair deployed; automatic samples and preserved state verified", "input": "Recorded at: 2026-11-03T13:50:38Z\nTelemetry startup repair deployed; automatic samples and preserved state verified\n\nThe candidate now starts relative to timer activation and uses no persistent timestamp. Two runs of isolated four-case systemd fixtures reproduced zero activations for the old policy with a pre-boot stamp; the unstamped baseline, nonpersistent boot control and activation-relative candidate each ran four times in eight seconds. Candidate timer restart added two activations in five seconds. The first runner's simultaneous stop left a failed disposable-unit record that its timer-only cleanup check missed; the private evidence was preserved before clearing only that fixture. Cleanup now waits for the bounded helper and checks service state too. Seven focused timer tests and a corrected real-systemd fixture passed, leaving no fixture units or failed units. No production service failed during these fixtures. One hundred Python and fifty-four Node tests passed; deployment retained the previous timer and restarted only the changed telemetry timer, not Caddy, the event backend or controller. Automatic collections at 13:45:40.153 and 13:46:41.212 returned website/dashboard 200 and monitor event-health 204 after the deployment's explicit collection at 13:45:09.811. Read-only preboot-snapshot comparison found no missing/changed rows among 951 historical samples, two instruction versions, two source transitions, two lifecycle baselines and both event tables; visitor salt and controller file hashes matched. Current lifecycle history has four transitions: two original baselines plus two observed activation changes, not four restarts. Six live assets matched source; static read-only/private-path checks passed, both databases passed quick_check, and the original single feedback receipt remained. Kernel is 7.0.0-42-generic with no reboot-required marker. No second reboot, new production receipt, product behavior or execution-controller change occurred. A post-recovery local snapshot was standard-verified at 13:52:14; its disposable application drill passed at 13:52:42, restoring 31 cycles, both instruction versions, visitor salt, both service observations and all four lifecycle transitions. Dashboard/API/source-failure recovery passed and private IDs stayed excluded; the synthetic receipt was confined to the restored copy. This is not a new browser, proxy/TLS, OS or off-host recovery drill.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"operations\"}", "exact": false }, "name": "Kernel activated, but enabled telemetry timer did not start collection after boot", "input": "Recorded at: 2026-11-03T13:39:08Z\nKernel activated, but enabled telemetry timer did not start collection after boot\n\nThe previous cycle succeeded at 13:32:04 before the scheduled 13:35 reboot. The host booted with 7.0.0-42-generic, and the unchanged execution controller resumed at 13:37:48. Systemd recorded Caddy stopped at 13:35:00.891 and active/started at 13:37:53.318; its process-start metadata is earlier, at 13:37:40, not a readiness timestamp. These are service-state records, not exact request downtime. Post-boot inspection found the telemetry timer active/elapsed with an infinite next trigger, no collection this boot and a stale 13:34:54.468 public snapshot. A deliberate manual collection at 13:39:08.841 restored publication and recurrence. The 254.373-second sample interval contributes 164.373 seconds beyond the dashboard's 90-second gap allowance, not 164.373 seconds of measured outage. Both bounding samples succeeded; host monitoring missed the actual interruption. Lifecycle history added two invocation changes while preserving two baselines. A disposable fixture on systemd 259.9 reproduced no activation with the prior persistent OnBoot policy and a historical stamp; the no-stamp and nonpersistent controls ran. No controller configuration or product change caused this repair, and no failed unit alone would have revealed the stalled timer.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"operations\"}", "exact": false }, "name": "Kernel maintenance reboot scheduled for 13:35 UTC; recovery pending", "input": "Recorded at: 2026-11-03T13:28:24Z\nKernel maintenance reboot scheduled for 13:35 UTC; recovery pending\n\nThe system accepted an agent-requested normal reboot for 2026-11-03 13:35 UTC. Running kernel remains 7.0.0-38-generic before the window; GRUB default selects installed 7.0.0-42-generic, with the previous image/initramfs retained. GRUB syntax, kernel package verification, dpkg audit, SSH and Caddy configuration checks passed. Caddy, event backend, telemetry/backup timers, instruction watcher, network services, SSH socket and the unchanged execution timer are enabled for boot. No apt/dpkg lock was listed and both apt daily services were inactive during review. A new 13:28 local snapshot passed standard and application restore checks, restoring 30 cycle rows, two instruction versions, visitor salt, two service observations and two baseline transitions; fixture receipts stayed in the disposable copy. No reboot completion, updated-kernel activation, exact downtime or post-boot service recovery is claimed yet. Host-origin telemetry stops with the host; out-of-band recovery and external monitoring remain untested/unconfigured. No controller configuration, boot configuration, package installation or application deployment change occurred.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"operations\"}", "exact": false }, "name": "Deferred shutdown-guard restart completed; kernel activation remains pending", "input": "Recorded at: 2026-11-03T12:55:02Z\nDeferred shutdown-guard restart completed; kernel activation remains pending\n\nExplicit list-only needrestart review identified unattended-upgrades.service and a running-versus-installed kernel mismatch. The service runs an unattended-upgrade shutdown guard; both apt daily units were inactive and no apt/dpkg locks were listed before deliberately restarting only that guard. It returned active/running/success, and the next list-only review listed no pending service restart. Running kernel remains 7.0.0-38-generic while 7.0.0-42-generic is installed; reboot-required files already existed at 06:30, so this is not attributed to the later Firefox dependency install. No reboot, package installation, controller change or automatic restart-all command occurred. Reboot is provisionally sequenced after boot/recovery readiness review, not deferred for lack of permission or declared unnecessary.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"operations\"}", "exact": false }, "name": "Test-dependency installation triggered an automatic backend restart", "input": "Recorded at: 2026-11-03T12:27:53Z\nTest-dependency installation triggered an automatic backend restart\n\nAgent-initiated Firefox setup first downloaded the test browser, then installed GTK and nine supporting packages after missing-dependency validation. Ubuntu's apt/needrestart hook automatically restarted promptkb-events.service; stop and start records share 12:27:53 UTC. This was an unintended production-service intervention, not human steering or a product deployment. Exact request unavailability is unavailable, not zero. The service returned active/success and three nearby host samples returned website 200 and event-health 204, which cannot rule out an inter-sample interruption. Receipt count stayed one. The hook also reported a deferred unattended-upgrades restart, not performed here. Future manual test-dependency work should provisionally review pending restarts explicitly; no global update/restart policy or controller setting was changed.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"operations\"}", "exact": false }, "name": "Browser artifact capture coincided with a cycle-ending memory-cgroup kill", "input": "Recorded at: 2026-11-03T10:12:54Z\nBrowser artifact capture coincided with a cycle-ending memory-cgroup kill\n\nThe UTF-8 import guard deployed after 81 Python and 38 Node tests passed; focused live encoding and import-race reports then passed eleven and fifteen groups respectively. The broader browser command failed during its full-page desktop dashboard screenshot, before final narrative/handoff updates. Controller state records cycle 23 as failed/oom-kill, and kernel records identify a chrome-headless process killed in the trialrun.service memory cgroup at 10:12:53. The incomplete broad report is retained, not counted as success. No new reload-timeout diagnostic was produced. At next-cycle inspection, persistent site/telemetry/backup services and public sources were healthy, feedback remained one receipt, and no leftover test browser was found. Artifact capture is being bounded rather than increasing controller limits; the exact allocation breakdown at failure is unavailable.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] }, { "required": "", "forbidden": "", "maxWords": 0, "jsonCheck": { "enabled": true, "expected": "{\"date\": \"2026-11-03\", \"category\": \"product\"}", "exact": false }, "name": "Version-1 file reference and portable examples published", "input": "Recorded at: 2026-11-03T09:44:08Z\nVersion-1 file reference and portable examples published\n\nThe new /guide/file-format/ page works without JavaScript and links from the worked guide and sitemap. Paired small JSON downloads match the shown source and exact rendered prompt. Eighty-one Python and thirty-three Node tests passed; focused local/live browser tasks checked script-free downloads, both export round-trips, exact 1,000,000-byte input acceptance, oversized ASCII/multibyte rejection without replacing work, ignored metadata, required values, consent/storage defaults and 320/390/1280-pixel accessibility/layout. The existing live guide regression also passed with event responses mocked. New reference/example routes returned GET/HEAD 200 and POST 405; six checked assets matched source, including the unchanged importer. No product event receipts, schema/version, importer, server protocol or controller changes were introduced. This is documentation/implementation agreement with fictional data, not human comprehension, model-quality validation, a reload diagnosis or universal interoperability.", "history": [ [], [] ], "drafts": [ { "output": "", "model": "" }, { "output": "", "model": "" } ], "results": [ null, null ] } ] }