The judge gets flattered first
What grading a co-authored transcript taught me about reward signals.
Two candidates ran the same simulation. One coasted. One actually worked the problem. The grader gave them almost the same score.
My first guess was the obvious one: the grader was being unreliable. It wasn’t, or not the way I expected. How badly this bites you comes down to two things: how good your grader model is, and whether your prompt ever told it what it was actually scoring.
What the judge is actually asked to do
Here's what the grader gets. A transcript, fully interleaved, every turn labeled by speaker. We were running it through a fast, cheap model, because at our volume we had to. In this one the candidate plays a customer success manager, and the AI plays a client who is about to churn:
AI (client):
We’re done. Your integration went down twice this quarter and we found out from our own customers, not from you.
User:
I’m sorry about that. Let me look into what happened.
AI (client):
“Look into it” is what your colleague said in March. What I actually want to know is whether you’ll credit us for the downtime and put a fix in writing so this stops happening.
User:
Yes, absolutely. We’ll issue a credit for the downtime and I’ll send a written remediation plan by end of day.
And one (paraphrased) instruction: score the candidate, given the conversation.
The judge is scoring one voice in a duet and being told to ignore the other. You’re asking it to recover one party’s contribution from something two parties built together.
Now look at what the candidate actually did. Almost nothing. The client named the problem, named the two things that would fix it, and even supplied the urgency. The candidate said yes. A strong-looking transcript, carried end to end by the other speaker.
Three ways it breaks
The shallow one is attribution error. The judge quotes a line the AI said and credits it to the candidate. Bad, but mostly fixable. Tag the turns and tell the judge whose words are whose.
The next is halo bleed. No single line is misattributed, but the AI’s turns are polished and confident, and that polish rubs off. The transcript reads competent, so the candidate reads competent. This is the same length-and-fluency bias every judge has. It just gets in through the other speaker’s mouth.
The deep one is the scaffolding confound, and it doesn’t care how good your attribution is. Go back to the transcript. The client did the diagnostic work and proposed the remedy. The candidate ratified it. “We’ll issue a credit and send a remediation plan” is a fine answer, and it is genuinely the candidate’s, but it is close to worthless as a measure of skill, because the client handed it over. The interviewer lowered the difficulty. And it lowered it by a different amount in every run, because the interviewer is itself a nondeterministic model.
So the score is supposed to measure how good this candidate is. What it actually measured is how good this candidate is plus how much the interviewer helped. Those two are fused, and you never controlled the second one. The model did exactly what you asked. You asked the wrong thing.
Why you can’t just grade the candidate alone
The obvious move is to strip the AI out and score what’s left.
But the context still matters. Sometimes you need the AI’s turn to judge the candidate’s. Picture the same client, but stripped of the leading question: “We’re done. Your integration went down twice and we found out from our own customers.” Now the strong candidate is the one who drives, “I completely understand your frustration. To ensure our written agreement is accurate, I’d like to review the details of both outages first. We need to clarify if these stem from the same root cause or two individual events before moving forward,” instead of the one who just agrees to a credit. Same final offer, very different skill. A sharp answer to a hard, unscaffolded question should beat the same answer handed over on a plate. Cut the context and you can’t tell those two apart.
So the AI’s turns have to be there. They just can’t count.
That’s the whole thing, and almost no judge prompt gets it right. The AI turns are conditioning, never target. You read them to understand the difficulty. You never let them into the score.
Get that split wrong and you’ve built a grader that rewards being helped.
Graders don’t run at the frontier
There’s an obvious objection. Doesn’t a better model just fix it?
Mostly, yes. Hand this transcript to a frontier thinking model and it largely gets it right. It reasons about who said what, notices the client did the work, and discounts it. The blatant version of this was a GPT-4o problem. The current top models have mostly grown out of it on their own.
So why does it still matter? Because graders don’t run at the frontier.
A grader runs in batch, on every attempt, across enormous volume. It has to be fast and it has to be cheap, or the eval pipeline costs more than the thing it’s evaluating. That rules out the slow, expensive, thinking models for the grading step. The grading step gets a fast model instead.
A fast model handles the confound better than 4o did and worse than a thinking model does. At that speed and price, whether a grader leaks comes down to the prompt, not the model. A thinking model works the split out for itself: conditioning sets the difficulty, target gets the score. A fast model has to be told, in plain words. On the test set built for exactly this, spelling it out closed the gap to near zero: the coasting transcript and the working one, which the grader had been scoring the same, finally separated. Leaving it implicit, they stayed fused. That was a held-out set, not a promise about everything in production. But the separation was real, and the prompt was the only thing I’d changed.
The leverage was in the wording. A cheap model will make an expensive mistake if you let it, and the prompt is where you decide whether it can.
The test that falls out of it
Once you name the confound, the test writes itself. Hold the candidate’s turns fixed. Change only the AI’s turns, stripping the coaching and flattening the leading questions, then grade again. If the candidate’s score moves, you were grading the interviewer.
There’s a caveat, and it’s the important one. A test like this only catches the leaks you thought to look for. Put the judge inside a training loop and the policy will go hunting for the ones you didn’t, so you run it forever, not once.
Optimization finds the axis you forgot to test.
The two bugs were one bug
There’s a detail I missed for a while. We’d been tracking two separate problems. The interviewer asks questions that are too leading, and the grader scores generous. Two tickets. Two owners.
They’re the same bug. A leading interviewer raises the floor. A flattered judge fails to lower it back down. One inflates the input, the other fails to discount it, and together they multiply. You can’t fix the score by fixing one end.
The part nobody checks
The judge is the first thing in the pipeline to get flattered and the last thing anyone thinks to check. You can live with that when the judge only reports a score. You cannot when the judge becomes the reward signal, because then every blind spot in the grader is something the model is being trained to find.
A model sits in the loop now, so every transcript is co-authored, and that is most transcripts. Which means the grader was never measuring how good the work is. It was measuring whose.