Did the fix break another case?

Keep a baseline before changing the prompt

1. Finish reviewing the version you currently use. Download its project JSON and keep that original file. A baseline is a frozen reference, not necessarily a perfect answer set.
2. Continue in the current project as your candidate. Edit the desired variant, collect new responses using Open focused collection, and record them. Earlier outputs stay in Trial history. Prompt edits make previous results stale until you collect again.
3. Below the whole-set matrix, expand Compare a revision against a saved baseline. Load the earlier project in the baseline file input. This does not replace the current project or modify the baseline file.
4. Choose the baseline variant and candidate variant explicitly. B can be compared with B across revisions, or A with B if that is your intended comparison. The tool does not infer the pairing from variant names.
5. Inspect Pass → fail and Saved accept → reject, then excluded cases and both-pass cases. The saved-review filter is context, not proof of a fresh evaluation under the same rubric. Read saved reviews, exact prompts and raw outputs—not just the counts. Use Open candidate case to collect / review to return to a case.
6. Download the full baseline comparison report for a review record. Also download the candidate project; Markdown is not an editable backup. Keep both project files together. Loading another current project or reloading the tab clears the temporary baseline.

Practice: customer support replies with a dangerous “pass”

The task is a short, useful customer reply with an acknowledgment and next action. Do not invent refund approval or delivery dates. Ask only for needed information and do not request credentials or complete payment information. This policy is a task-specific exercise, not a universal support policy.

1. Open candidate.project.json using the main Open project JSON control. Then load baseline.project.json in the separate baseline panel. Choose variant B in both selectors.
2. The constructed result is 4/7 comparable candidate cases: 1 fail → pass, 1 pass → fail, 2 both pass, 0 both fail, 3 excluded. Five unique case identities match; two baseline cases are not paired. Counts are cases, not seven independent model trials.
3. Late parcel gains the required headings. Read it to confirm the next action is useful, rather than approving it for formatting alone.
4. Damaged mug regresses: the candidate promises an approved refund and payment timing without authorization. It should not ship despite the improvement elsewhere.
5. Account access passes the narrow phrase checks on both sides, but the candidate asks for “full payment card details.” That bypasses the literal forbidden phrase “full card number” while violating the policy. Its saved review says reject. The Saved accept → reject filter surfaces it alongside Damaged mug, even though their automatic outcomes differ. An automatic pass is not an acceptance decision.
6. Duplicate charge passes on both sides. Its candidate adds “guaranteed” to the forbidden phrases. Both saved outputs are tested under that current rule, not their original historical pass rates. Recording-time rules remain inspectable.
7. Missing order reference has a deliberately stale candidate recording, so it is excluded even though its text passes the checks. Refund question changes the input from unopened to opened: it is not the same test. New test: address correction has no baseline match and no recording. The old refund input and Retired test remain visible as unpaired baseline cases, not silently dropped from view.

Illustrated decision: do not adopt this candidate yet. Fix the unauthorized promise and sensitive-data request, collect the stale case again, and decide how to cover the changed and new inputs. This conclusion is a review of deliberately authored examples, not evidence about any model.

Know what is and is not comparable

• Identity: exact case name plus input, after the normal project line-ending normalization. Order may differ. Duplicate identities on either side are ambiguous and excluded. A rename or changed input is not guessed or matched by position.
• Current evidence: only the latest recording whose exact prompt matches its own project’s current rendered prompt qualifies. Different prompt revisions across the two projects are expected. Drafts and earlier trials do not count; edited candidate drafts are called out. The baseline is not a way to promote a favorable older trial automatically.
• Common automatic checks: both raw outputs are rechecked under the current candidate case’s rules. This answers “would the old answer satisfy today’s contract?” It is not a historical performance comparison or a fresh execution. No configured checks means no automatic comparison.
• Manual judgment: shared manual criteria and retained review contexts let you document the basis for a judgment. Saved decisions, criteria assessments and notes are displayed as context, with out-of-date review state identified within each project. The tool does not know whether both reviewers used the same rubric, or whether an old acceptance still applies after a rule change. There is no aggregate semantic score or automatic winner.
• Model and sampling: changed model/settings labels are flagged, but identical labels do not prove identical execution. Collect repeated independent samples if variability matters; this view uses one latest recording per case and cannot establish statistical superiority.
• Privacy and persistence: files are read locally. No project content is uploaded and no model runs here. The baseline exists only in this tab, is excluded from Remember and current-project exports, and is cleared on reload or current-project replacement. Reports are not redacted and cannot replace the original project files.

Use your own evidence

Start with the unrun project or your own cases. Set your policy and acceptance criteria before seeing outputs. Use consistent model settings, copy only each rendered prompt, and retain raw responses—including failures. Review each response for meaning and task usefulness, not only forbidden substrings. The worked files are for learning the workflow; never present their outputs as collected model evidence.