How to triage a noisy model evaluation
When an evaluation score moves unexpectedly, do not start by changing the prompt. First separate data problems from model problems. Check whether examples changed, labels drifted, graders became inconsistent, or a small number of cases dominate the score.
Group failures by intent and error type, then inspect representative examples from each cluster. Re-run a stable baseline on the same data. If the baseline also moves, the evaluation changed. If only the candidate moves, investigate the model or prompt.
The goal is not to explain every failure individually. It is to find the smallest number of causes that account for most of the regression.
- Category
- Research
- Platform
- Web