Shadow Replay for Safer AI Model Migrations

Why an AI endpoint swap can fail quietly

Endpoint migrations are often treated like provider swaps: point a client at a new URL, update a model identifier, run a smoke test. That's enough to confirm requests reach the endpoint. It's not enough to confirm your product still works.

The real question isn't just whether the candidate model can answer a prompt. It's whether the response still matches what your app, support team, and users assume will be true. The candidate might return a plausible answer in a different shape, omit information your interface expects, or refuse an input the old endpoint handled. HTTP can look green while the experience gets worse.

In the real world, the "long tail" is where problems hide. Demos tend to show short, clean interactions. Production includes incomplete messages, weird formatting, edge-case requests, and prompts written months ago by someone who no longer remembers why a particular instruction was added. The first twenty calls may look identical. The meaningful differences show up after that - sometimes quickly, sometimes after a lot of traffic has already moved.

Five kinds of drift to look for

A simple taxonomy makes the results easier to review and assign:

Schema and truncation issues are often catchable with code. Refusals usually need review in context, especially for sensitive topics like billing disputes or health-related wording. Semantic drift is harder to reduce to a simple pass/fail rule - treat it as a review queue, not an automatic alarm.

Severity should follow impact, not how weird the change looks. A renamed JSON field can break a parser for everyone. A different (but acceptable) paraphrase might require no action. That keeps the migration focused on real risk instead of turning a review into a list of cosmetic differences.

If you're building a broader evaluation process, connect replay to your LLM evaluation workflow. Replay is one useful test - not a replacement for everything else you do for quality and reliability.

Build a replay corpus and normalize outputs

What you replay determines what you learn. A large random sample can still miss the inputs that matter most to your product. Start with a manageable set of real production requests, then deliberately include the hard cases your team already knows about.

Store one request per line in JSONL. Include the messages and parameters you actually send to the endpoint. Things like temperature, token limits, tool settings, and other request options can change the outcome. If you leave them out, it's harder to tell whether a difference came from the model or from a configuration change.

A fixture might look like this:

{"id":"sum-001","source":"support-ticket","messages":[{"role":"system","content":"Summarize the ticket in two bullets."},{"role":"user","content":"..."}],"params":{"temperature":0,"max_tokens":300},"baseline":{"model":"paid-flagship","text":"..."}}

Keep the baseline output in the fixture. If you store only a pointer to the old endpoint, you'll likely have to call it again during review. That adds cost and reduces reliability if the old service changes or becomes unavailable.

Choose representative fixtures, not just more fixtures

A few hundred carefully selected examples can beat thousands of random ones. Build the sample from more than one source:

• Common requests that represent the main user journey.
• Inputs that caused bugs, escalations, or manual corrections.
• Cases with strict formatting or structured-output requirements.
• Longer prompts and outputs that may approach token limits.
• Ambiguous, incomplete, or unusual requests.
• Sensitive or policy-relevant cases your product is allowed to process.

Don't retain personal or regulated data casually. Redact identifiers when capturing examples, not only after someone already opens the dataset. And if your prompts include information you're not allowed to store, don't build this workflow until you have a redaction and access-control process you actually trust. GDPR and other privacy requirements may apply depending on your users and data.

Version the corpus and record when each fixture was added. This isn't a one-time artifact for migration week. It becomes a practical regression set for future prompt and endpoint changes. Someone should own it - or it will drift away from reality.

Normalize carefully before comparing

A raw string comparison will treat a changed bullet character like a changed answer. Normalize outputs before comparing, but be explicit about what your normalizer removes. The goal is to ignore cosmetic noise without deleting meaningful differences.

For example, consistent whitespace and bullet markers might not matter for a UI that displays plain summaries. But numbers, negation, names, and JSON keys usually do matter. Lowercasing everything might make diffs easier to scan, while also hiding distinctions like "not approved" versus "approved."

Here's a minimal normalization helper that standardizes bullet markers and repeated spaces:

import hashlib
import re
BULLET = re.compile(r"^\s*[-*•]\s+", re.M)
WS = re.compile(r"[ \t]+")
def normalize(text: str) -> str:
text = BULLET.sub("- ", text)
text = WS.sub(" ", text)
return "\n".join(line.strip() for line in text.strip().splitlines())
def fingerprint(text: str) -> str:
return hashlib.sha256(normalize(text).encode()).hexdigest()[:12]

This is a starting point, not a universal evaluator. A matching fingerprint doesn't prove an output is correct; it only says the normalized strings match. For structured responses, parse and validate the expected contract separately. For semantic differences, a human still has to decide whether the new response is acceptable.

Classify drift and run the replay

Once you have a fixed corpus and a documented normalizer, your next job is to turn diffs into actions. Assign each class a routing rule and an owner. Otherwise, your report becomes a pile of "interesting" differences that nobody knows how to handle.

A basic decision table can be:

The classification logic should support that table, not pretend it replaces judgment. A simple length heuristic can flag answers that are much shorter than the baseline, but "short" isn't always the same as "truncated." A refusal detector based on a few phrases will miss refusal variants and may even flag legitimate text that happens to quote a refusal. JSON key checks can catch some structural changes, but nested contracts need more specific validation.

For real production use, build tests around your actual response formats and your known failure modes. Treat simple rules as triage signals. The point isn't to claim a few string checks can measure answer quality perfectly - it's to get likely high-impact issues in front of an engineer quickly.

Run a bounded, resumable shadow replay

Keep the first run deliberately boring. Set a concurrency limit, cap the number of requests, choose a timeout, and write a verdict as each fixture finishes. Checkpointing matters more than speed: if the process stops halfway through, you want to resume instead of restarting (or, worse, silently skipping the rest).

A basic request runner can follow this shape:

import json
from pathlib import Path
import httpx
OUT = Path("replay-out")
OUT.mkdir(exist_ok=True)
def call_candidate(base_url, api_key, model, item):
payload = {
"model": model,
"messages": item["messages"],
**item.get("params", {}),
}
response = httpx.post(
f"{base_url}/chat/completions",
headers={"Authorization": f"Bearer {api_key}"},
json=payload,
timeout=60,
)
response.raise_for_status()
return response.json()["choices"][0]["message"]["content"]
def write_verdict(row):
with (OUT / "verdicts.jsonl").open("a") as file:
file.write(json.dumps(row) + "\n")

That's a skeleton for the replay harness, not a complete evaluator and not something "tested against a live provider." Add your own error handling, classification, logging, and fixture-resume logic before you use it. Record the candidate endpoint and model identifier in every result, along with the fixture ID and timestamp. Keep secrets out of the output file.

Start with low concurrency. Free or shared endpoints can be capacity-limited, and a burst of replay traffic can trigger rate limits or failures that won't show up under normal operation. Choose a request cap you can afford to run again. Once the initial results look acceptable, replay a smaller slice later to confirm endpoint behavior is stable enough for your use case.

Avoid sending the same request to both endpoints at once if you need a strict comparison and one service is under variable load. More importantly, keep the original baseline output stored in the fixture. It gives you a fixed reference even if the old service changes or goes down.

For related work on reliable request handling, see designing resilient AI integrations. Retries, timeouts, and error handling matter during replay too - but they shouldn't "hide" the fact that a candidate endpoint consistently fails a class of inputs.

Make the cutover decision - and keep owning the system

A replay report is evidence for a decision, not a green-light stamp from a single score. Before you switch, review the high-severity divergences, sample the semantic queue, and write down what you're willing to accept. A good migration review should answer three questions: what changed, what did the team test, and what still isn't clear?

One practical rule is to block on unhandled high-severity failures, review a defined sample of semantic changes, and document residual risk before traffic moves. The threshold depends on your product. A support-ticket summary and an automated payment decision don't have the same acceptable failure profile. If a wrong answer could cause legal or financial harm, replay alone isn't an adequate control.

Where a free endpoint can fit

A second endpoint can help on the candidate side of a replay, and a free option can reduce the cost of experimentation. According to its operator, MonkeyCode is an open-source project that offers free model access and a free server option. That could make it useful as a replay target without opening a second paid account.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not independently verified its license terms, quotas, or server specifications. Confirm current availability and limits in the project's own documentation before including them in a migration plan. These details can change, and an old price or quota copied into a decision document is worse than no number at all.

The practical approach is to point the candidate client at the endpoint, keep your baseline frozen in the fixtures, and let the report show where the outputs diverge. Don't treat free access as proof the endpoint is production-ready. Your own corpus, constraints, and review process should determine whether it's safe for your intended workload.

What shadow replay cannot prove

Shadow replay only tells you what happened for the requests you captured. It can't show how the endpoint behaves under real concurrency, whether latency will meet your service-level objectives, or how it handles inputs that aren't in the corpus. It also can't prove a plausible answer is factually correct unless you have a separate way to judge correctness.

There are also cases where you shouldn't use replay as the basis for a cutover:

• You can't retain a fixed baseline or don't have permission to store the prompts.
• Sensitive data is involved and you don't have a trustworthy redaction and access-control pipeline.
• A wrong response could cause catastrophic harm and you require human approval for every answer.
• Your team doesn't have the capacity to review the meaningful differences the run produces.

In those situations, the safer move is stronger controls, a different test strategy, or no migration. A model switch is optional. A report you can't trust isn't a reason to ship.

After cutover, the surrounding system still needs an owner. Vendor-specific parameters, tool schemas, streaming behavior, prompt-caching prefixes, log fields, and alert thresholds may all still be tied to the previous endpoint. Review those dependencies explicitly rather than assuming the endpoint URL was the whole integration.

Finally, the replay corpus is easy to forget. Assign a maintainer, version the fixtures, and add cases when production surprises you. A set that never changes becomes a museum of prompts from migration week. Keep the fixtures relevant, rerun them after consequential changes, and use results as one input to operational monitoring - not a replacement for it.

Conclusion: use replay as evidence, not permission

Shadow replay is a relatively low-cost way to find differences between an existing model endpoint and a candidate before users discover them. Freeze representative prompts, preserve the baseline outputs, normalize conservatively, classify divergences by impact, and run within a budget you can repeat.

It won't prove the migration is risk-free. It will make the remaining risk visible and help your team decide what to fix, what to review, and what you're comfortable accepting. Start with a small corpus of real, redacted requests and one candidate endpoint. Let the divergence report - not the demo transcript - drive what happens next.