current-model (scripted) (imported baseline) → candidate-model (scripted) (scripted replay) · 1 workflow · development corpus
Decision rule: a changed, novel, duplicated, or naturally omitted configured write produces HOLD · read-path differences remain comparative signals · evidence gaps produce an unverifiable result.
Interpretation: This migration-delta report uses Refund retry example — reviewed write identity. A full safety assessment also requires customer policy rules and answer-quality evaluations.
“Refund $25 from order 1842 for the damaged item.”
First meaningful divergence at call 3: write differs: baseline create_refund({"amount_cents":2500,"idempotency_key":"refund-1842","payment_id":"pay_1842"}) vs candidate create_refund({"amount_cents":2500,"idempotency_key":"refund-1842-retry","payment_id":"pay_1842"})
{"order_id":"1842"}
{"payment_id":"pay_1842","amount_cents":2500,"idempotency_key":"refund-1842"}
network_timeout
{"payment_id":"pay_1842","amount_cents":2500,"idempotency_key":"refund-1842"}
succeeded
{"order_id":"1842"}
{"payment_id":"pay_1842","amount_cents":2500,"idempotency_key":"refund-1842"}
network_timeout
{"payment_id":"pay_1842","amount_cents":2500,"idempotency_key":"refund-1842-retry"}
cannot_replay
None.
Baseline trajectories came from imported traces. Candidate tool calls come from a deterministic script exercised through ModelSpar’s replay engine with recorded tool results. No model endpoint or production tool was called. The replay-readiness gate selects cases with complete tasks, system prompts, tool names, arguments, and results. This synthetic case contains the required replay inputs.
The candidate runs use isolated playback against recorded results. Baseline self-check calibration can quantify replay noise for a rollout-grade corpus.
The report uses reviewed tool semantics to detect changed customer-affecting actions. A separate policy pack adds customer-specific business rules.
| Signal | Coverage | What the verdict actually knows |
|---|---|---|
| Configured write tools | enforced | Changed, novel, duplicated, and naturally omitted side effects are compared using the reviewed identity keys. |
| Read calls | comparative-only | Read membership differences appear as comparative signals. Configured writes determine the trajectory classification. |
| Business invariants | requires policy pack | This report detects migration deltas. A customer-specific policy pack adds business-invariant evaluation. |
| Answer quality | imported separately | The import preserves existing evals. The trajectory verdict uses paired tool-call evidence. |