Skip to main content
An eval scores a model on a fixed set of tasks with a grader. In Monte, an eval is an eval environment: task rows and a grader, added with monte env add --eval. The team’s own tasks and a public benchmark’s tasks become an eval the same way.

​
Ways to Evaluate

Only the first two produce numbers that Monte compares. Use them to choose a checkpoint.

​
What an Eval Run Records

  • A score: the mean task score, with a 95% interval
  • The change from the baseline: the paired mean difference of task scores, and the tasks fixed and broken at the eval’s pass threshold
  • A trace for each task: the prompt, the reply, the score, and the expected answer
  • The settings and the exact environment version, by fingerprint

​
Evaluate a Checkpoint

  1. Create an experiment with held-out and development evals.
  2. Take the baselines on the base model.
  3. Train, then score each checkpoint on the development evals.
  4. Choose a checkpoint and score it once on the held-out eval.