TL;DR
This guide develops Evaluating RL Agents: Trajectory Evals, Tool-Use Metrics, and LLM Judges from its central engineering problem to practical design choices and limitations. It synthesizes the roadmap topics without claiming new experimental results. Reported findings belong to the cited sources.
Target searches: “agent evaluation”, “how to evaluate LLM agents”
Why This Topic Matters
Modern LLM and agent training turns familiar reinforcement-learning ideas into systems problems: actions may be tool calls, rewards may come from fallible graders, and trajectories may span many partially observed steps. The classical foundation is safe and constrained RL; this article focuses on what changes in the newer setting.
The Core Ideas
1. Why final-answer accuracy is insufficient for agents — right answer via wrong process will not generalize
2. Trajectory metrics — tool choice correctness, argument correctness, decision ordering, reasoning quality per step
3. System metrics — tokens per task, latency, tool-call frequency, failure-recovery rate, cost
4. Outcome metrics — task completion, and who decides
5. The framework landscape: DeepEval, Braintrust, Arize Phoenix, OpenAI Evals, RAGAS, LangSmith, Galileo — what each is actually for
6. The production loop worth stealing: label every production turn with a per-turn classifier, then route those labels into your eval set, your fine-tuning data, and your RL reward terms
7. Eval variance — how many runs before a difference is real
Practical Checklist
- Define the state, action, termination condition, and reward before choosing an optimizer.
- Log raw trajectories and every reward component, not only the aggregate score.
- Separate training measurements from held-out evaluation and adversarial tests.
- Treat generated code, tool calls, and environment actions as untrusted.
- Record model, framework, benchmark, and grader versions so comparisons remain reproducible.
Limitations
This area changes quickly. Framework support and benchmark leaderboards are time-sensitive, while many recent methods are preprints rather than settled practice. The article describes mechanisms and reported evidence; it does not present an original training run.
Key Learnings
- Why final-answer accuracy is insufficient for agents — right answer via wrong process will not generalize
- Trajectory metrics — tool choice correctness, argument correctness, decision ordering, reasoning quality per step
- System metrics — tokens per task, latency, tool-call frequency, failure-recovery rate, cost