TL;DR
This guide develops Verifiers, Rubrics, and LLM Judges: Building Graders That Don’t Get Gamed from its central engineering problem to practical design choices and limitations. It synthesizes the roadmap topics without claiming new experimental results. Reported findings belong to the cited sources.
Target searches: “LLM as judge”, “reward function design LLM”
Why This Topic Matters
Modern LLM and agent training turns familiar reinforcement-learning ideas into systems problems: actions may be tool calls, rewards may come from fallible graders, and trajectories may span many partially observed steps. The classical foundation is model-based RL; this article focuses on what changes in the newer setting.
The Core Ideas
1. The grader taxonomy and what each is appropriate for
2. Writing deterministic verifiers: normalization, equivalence classes, the long tail of “technically correct” answers
3. Rubric graders — decomposing a judgment into checkable sub-criteria
4. LLM-as-judge: position bias, verbosity bias, self-preference bias, and the fact that the judge is itself a model that can be persuaded
5. The rule that should govern the choice: use deterministic checks for anything mechanical, reserve judgment-based grading for what genuinely requires judgment
6. Grader validation — how do you know your grader is right? Agreement with human labels, and adversarial testing of the grader itself
7. Composing multiple graders (links to A6)
Practical Checklist
- Define the state, action, termination condition, and reward before choosing an optimizer.
- Log raw trajectories and every reward component, not only the aggregate score.
- Separate training measurements from held-out evaluation and adversarial tests.
- Treat generated code, tool calls, and environment actions as untrusted.
- Record model, framework, benchmark, and grader versions so comparisons remain reproducible.
Limitations
This area changes quickly. Framework support and benchmark leaderboards are time-sensitive, while many recent methods are preprints rather than settled practice. The article describes mechanisms and reported evidence; it does not present an original training run.
Key Learnings
- The grader taxonomy and what each is appropriate for
- Writing deterministic verifiers: normalization, equivalence classes, the long tail of “technically correct” answers
- Rubric graders — decomposing a judgment into checkable sub-criteria