Benchmark tasks and evaluation infrastructure for LLM training
RLHF benchmark tasks with reference solutions and automated graders, including multi-step agentic tasks, plus the harness that scores runs.
Problem
Model teams need realistic, verifiable tasks, including multi-step agentic ones, with grading that does not depend on a person reading every transcript.
What I built
- Benchmark tasks with reference solutions and automated graders.
- Agentic tasks where the model has to use tools across several steps to reach a checkable end state.
- The evaluation harness that scores runs, aggregates results and surfaces failure modes.
Outcome
Task content is not public. I’m happy to talk through the evaluation approach and how the graders were validated.