Eval Before Switch

Do not switch a production model on benchmark scores alone. Public benchmarks are contamination-prone and measure the wrong workload.

Required before a switch:
1. Build or reuse an eval set of at least 20 tasks drawn from real closed issues or real traffic, with objective pass criteria.
2. Run both the current and candidate model against it with the actual production prompts and tooling, not simplified versions.
3. Compare on pass rate, cost per completed task, and time to first token. All three, not just quality.
4. Check tokenizer differences. The same text can cost meaningfully more or less on a different tokenizer.
5. Report regressions per task, not just in aggregate. An average that holds while three specific task types collapse is the usual shape of a bad switch.

If the candidate is not clearly better on the metric that matters, do not switch.