How to triage a noisy model evaluation
When an evaluation score moves unexpectedly, do not start by changing the prompt. First separate data problems from model problems. Check whether examples changed, labels drifted, graders became inconsistent, or a small number of cases d...
