Skip to main content
Is your current model the right one for the job? Model Analysis replays your team’s real requests against cheaper or better models, so you can compare them with evidence from your own workload before you switch. Results are scored by category agreement for classification answers or by a blind LLM judge for open-ended responses.

​
Run a comparison

1

Freeze a dataset

Pick real sessions captured by your proxy and freeze them. Freezing makes no provider calls, so you get a stable, repeatable test set. Each dataset uses one scoring method: category agreement or a blind LLM judge.
2

Prepare a full run

Open Prepare full run on any frozen dataset. The panel shows the source baseline and, for judge datasets, the frozen judge model.
3

Choose target models

Search the model catalog and tick the targets you want to test.
4

Check the preflight

The compatibility preflight makes no provider calls. It shows how many candidate and judge calls the run needs and the maximum attempts before you confirm.
5

Review, filter, retry

Follow progress and results, filter by state or target model, and retry any failed cases. Open a case for its evidence panel and switch between Changes, Side by side, and Raw diff.
Before each replay the request is normalized: the model is replaced, streaming is turned off, unsupported fields are dropped, and credentials are removed. The replay then goes through your governed enterprise proxy, so every comparison follows your existing controls.

​
Reading the results

Model Analysis scoreboard: five classification sessions replayed on Moonshot Kimi K2.6 with 5 of 5 agreement
Category agreement. Five classification sessions replayed on Moonshot Kimi K2.6 all matched the original labels: 100% agreement.
Larger classification dataset with 75 of 100 sessions in agreement and 9 label-extraction failures listed under compatibility and failure reasons
At scale, failures included. A 100-session dataset reached 75% agreement. Nine cases couldn’t be scored because label extraction failed; they appear under Compatibility and failure reasons and stay in the denominator, so the result isn’t inflated.
Blind LLM judge scoreboard comparing summaries from the current model with Azure GPT-5.6 Luna, target won 2 of 3
Blind judge. Summaries from the current model were compared with Azure GPT-5.6 Luna. The judge never sees which model wrote which answer. The target won 2 of 3 sessions, a 67% win share, and the verdict lists strengths and weaknesses for each side.
Prepare full run panel with a target model ticked and the compatibility preflight enabled

​
FAQ

No. Freezing only snapshots sessions your proxy already captured. Provider calls happen only after you confirm a run, and the preflight shows the call count first.
Use category agreement when the original answer is a label or classification. Use the blind LLM judge for open-ended responses like summaries.
No. Cases that can’t be scored are listed under Compatibility and failure reasons and stay in the denominator. You can retry them from the results view.