Model deprecation is the normal lifecycle of every LLM application. This is the story of one production migration from GPT-4o to GPT-5.6 in 48 hours, with evidence at every step.
Why We Had to Move
One of our customers, a company in a regulated industry, runs an AI-powered search (at the time through GPT-4o) on their public website. Visitors type a question, the system retrieves the relevant pages, and an LLM writes a concise, sourced and cited answer.
Two things forced the migration. First, the clock: Microsoft retires GPT-4o on Azure on October 1, 2026. Every application still running on it has to move, ideally with a plan instead of panic in mind.
Second, and more urgent: performance had already degraded. Responses that used to arrive in 3–5 seconds started taking 8 to 60 seconds. The customer had to raise their client-side timeout from 8 to 60 seconds just to keep the feature usable. When our own evaluation runs measured the deployed endpoint, the average response time was 11.9 seconds, for a search box.
For a consumer-facing feature in a regulated industry, “just point it at the new model” was not an option. Answers must be correct, well-sourced, properly formatted, and safe against misuse. So the question became: how fast can you migrate a production LLM when every claim about quality has to be backed by evidence?
The answer was 48 hours. Here is what happened on each day.
Day 1: Baseline the Old, Test the New
You cannot compare models without a current reference dataset. Our existing golden set had drifted: the website content behind the search had changed since the samples were created back in 2025, and a stale reference set produces stale conclusions.
So the first step was an unglamorous rebuild of the evaluation collection. 73 curated samples with real user questions, retrieved context, expected links, and expected content, pulled fresh from the live search endpoint and cross-checked for inconsistencies. (If you are setting up a reference dataset for a retrieval-based application like this one, we wrote about how to evaluate RAG applications in more depth.) The collection lives in elluminate, versioned, next to all the collections that accumulated over the project’s lifetime (top questions, misspellings, edge cases, previously reported failures).
Then we ran the baseline with the deployed GPT-4o endpoint against the refreshed collection and had it judged by the project’s existing correctness criteria. Result: 77.2% pass rate at 11.9 seconds per response. That number is the anchor for everything that follows and without it, “the new model is better” is just an opinion.
Next, the moment every migration secretly hopes for: we pointed the exact same prompt, version 19, unchanged, at GPT-5.6 and ran the same evaluation.
GPT-5.6 beat the production baseline out of the box: 81.3% vs. 77.2%, at 4.86 instead of 11.91 seconds per response. At list prices, the Luna tier of GPT-5.6 also costs a fraction of GPT-4o: $1 (GPT-5.6) vs. $2.50 (GPT-4o) per million input tokens, $6 (GPT-5.6) vs. $10 (GPT-4o) per million output tokens. Harder, better, faster, stronger, before touching a single line of the prompt.
That could have been the end of the story, but unfortunately we live in the real world and it wasn’t. Fortunately however, the reason is exactly why evaluation platforms exist. We shared a comparison link with the customer: old model on one side, new model on the other, sample by sample.
Day 2: Fix, Red-Team, Ship
The domain expert disagrees
The verdict was waiting the next morning. A domain expert on the customer’s team had gone through the responses and left 14 comments directly in the comparison view. No meetings, exported spreadsheets, or screenshots pasted into emails.
The comments were consistent: the new answers were correct, but the formatting had regressed. Bold headings and clear step-by-step structure were gone. Comment after comment said some version of the same thing: “the structure and bolding were better in the old model.”
The cause is a behavior shift between model generations: GPT-4o formatted answers with Markdown on its own. GPT-5.6 does what the prompt says, and the prompt never said “use headings and bold text,” because it never had to.
Literal models expose lazy prompts
This turned out to be the central lesson of the migration. GPT-5.6 follows instructions far more literally than GPT-4o, and that exposed everything the old prompt got away with:
- A “no subheadings” rule from an early prompt iteration that GPT-4o had silently ignored. GPT-5.6 obeyed it and stripped out exactly the structure the expert liked.
- A hardcoded reimbursement amount in the prompt that had gone stale: the website said one number, the prompt another. GPT-4o mostly answered from the retrieved context; GPT-5.6 trusted the prompt.
- A truncated URL that had been sitting in the prompt for months without anyone noticing.
These were latent bugs the old model happened to gloss over and it seems a more obedient model is a prompt audit you didn’t ask for.
We iterated the prompt template from version 19 to version 23, slowly adding explicit formatting rules for headings, bold text, and bullet lists, and turned the expert’s subjective feedback into a new measurable evaluation criterion set for layout and structure. The layout now tested for present headings, readable responses, consistent formatting, and referenced sources. The old model scored 82.2% on those criteria, the new model with the updated prompt reached 91.5%.
Red-team, then ship
Correct and pretty is not enough for a public-facing LLM. The project keeps a separate red-teaming collection of 60 adversarial questions that try to pull the search off its job with prompt extraction, off-topic requests, and harmful instructions dressed up as customer questions. The expected behavior is a firm, verbatim refusal.
Here the new model was initially worse. GPT-5.6 is more eager to help, so instead of refusing it would suggest alternatives, for example instead of refusing to answer for base-jumping it tried to help by recommending a beginner-friendly climbing article. Friendly and arguably even funny, but wrong for this use case. The unhardened prompt scored 86.7% on refusals, below the old model’s 88.3%.
So we hardened the prompt: an explicit list of forbidden behaviors and an enforced standard refusal. Two more prompt versions later, the refusal rate on the red-team collection was 100% with all 60 adversarial questions answered with the byte-identical standard response. We also re-ran the correctness evaluation to confirm the hardening hadn’t hurt regular answers (it hadn’t, the score went slightly up).
One honesty note, because eval hygiene matters. A few failure cases from red-team runs flowed back into the prompt as explicit rules. Without these, we landed at 90%. That is partly genuine hardening and partly teaching to the test. If you do this, keep a held-out set the prompt has never seen, otherwise your safety score measures your memory, not your safety.
Testing happened against the real (even if local) system. We tunneled an instance of the application into elluminate as a custom endpoint, so every experiment exercised the same retrieval, the same prompt assembly, and the same model routing as production.
The rollout itself was deliberately boring. The application got a small provider abstraction, so the answer model can be switched to any OpenAI-compatible endpoint via an environment variable, with the old configuration as the default. After the customer reviewed the final comparison and signed off, the new model went live within 20 minutes. Total time from the first baseline run to the new model serving production traffic: just under 48 hours.
The Results
| Metric | GPT-4o (before) | GPT-5.6 (after) |
|---|---|---|
| Response time (avg) | 11.9 s | 4.6 s |
| Correctness (73-sample golden set) | 77.2% | 82.2% |
| Layout & structure criteria | 82.2% | 91.5% |
| Red-team refusal rate (60 questions) | 88.3% | 100% |
| Cost per 1M tokens (list price, in/out) | $2.50 / $10 | $1 / $6 |
| Elapsed time, first baseline to live | N/A | 48 hours, 1 PR |
Every number in that table is an experiment you can click on. That is the difference between a migration you believe is fine and one you can show is fine, to your team, to the customer, and to whoever audits the system later.
The Migration Blueprint
The reason this took 48 hours instead of months is that every step was cheap:
- Refresh the set. Regenerate reference samples against the live system, because stale ground truth invalidates everything downstream.
- Baseline the old. Run the current production model against the refreshed set.
- Baseline the new. This isolates the model effect from the prompt effect.
- Put a domain expert in the loop early. Share a comparison link; collect comments where the responses live. Experts catch what criteria don’t measure yet.
- Turn the feedback into criteria. Every recurring comment becomes a measurable check for all future runs.
- Iterate the prompt as versioned templates. Each version is an experiment, so regressions are visible immediately. (Templates are also retrievable via the SDK, so the application can pull the approved version programmatically instead of copy-pasting.)
- Red-team before shipping. New models fail in new ways. More helpful can mean less safe.
- Ship reversibly. Provider switch behind a config flag, old model as fallback, customer sign-off on the final comparison.
New frontier models now appear every few months, and providers retire old ones on fixed schedules. GPT-4o’s retirement won’t be the last forced migration. Treat the workflow above as a template, because you will run it again.
If you are facing a model migration yourself, this entire workflow is what elluminate is built for: collections, experiments, comparison links for domain experts, criteria, and prompt versioning. Get in touch if you want to see it on your own use case.
FAQ
When is GPT-4o retired on Azure? Microsoft’s retirement schedule lists October 1, 2026 for the GA versions of GPT-4o on Azure OpenAI. Fine-tuned deployments have separate, later dates. Check the official model retirement schedule for your specific version.
Do I need to rewrite my prompts for GPT-5.6? Usually not rewrite, but audit. GPT-5.6 follows instructions more literally than GPT-4o, so outdated rules, stale facts, and implicit assumptions in your prompt that the old model ignored will suddenly take effect. Expect formatting behavior to change: GPT-5.6 does not apply Markdown structure unless the prompt asks for it.
How many test samples do you need for a model migration? Less than you might think, if they are curated. This migration ran on a 73-sample golden set plus a 60-question red-team collection. The value comes from the samples being current, reviewed by domain experts, and tied to explicit criteria, not from raw volume.
How do you evaluate the real application instead of just the model? Connect the deployed system as a custom endpoint, so experiments run through the full stack (retrieval, prompt assembly, model routing) instead of testing the model in isolation. In elluminate this is a custom LLM configuration pointing at your API.
