Skip to main content
monte eval scores one model on one eval of an experiment and records an eval run.

​
Score a Checkpoint

The second argument names the eval. It can be left out only when the experiment has one development eval. --remote runs the eval on a GPU instance. --smoke scores 3 tasks to test the setup and never counts as a result. Eval Commands lists every flag.

​
Follow, Inspect, or Stop a Run

monte run list prints each run with its alias, such as eval@200, and its score. monte run show prints the target, the settings, and the score of one run. monte trace list prints the per-task results. Observability Commands lists every flag.

​
Eval Settings

Each eval keeps the settings pinned when the experiment was created. An eval environment can declare its own values. Others take the default. No flag changes them. Other settings need a new experiment.

​
Read the Result

Each line shows the baseline, the newest score, and the change. A score is the mean task score. The change is the paired mean difference of task scores, with its 95% interval. Fixed tasks were below the eval’s pass threshold at the baseline and reach it now. Broken tasks reached it and fall below it now. A change is clear when it stays above zero after the interval is taken off. The Console shows each run under Evaluations, with a trace for each task.
Both runs score the same n tasks, so Monte compares them task by task. d is the list of per-task differences: the newest score minus the baseline score.
The interval is 95%. Monte compares only runs on the same eval, in the same experiment, from the same base model.

​
Regrade a Run

monte grade scores the replies of an eval run again with the grader of another experiment, without serving the model:
Fix a Grader and Regrade shows when to use it.

​
Choose a Checkpoint

The best development change over many checkpoints overstates the gain, so the held-out score is the result to report.
  1. Choose the checkpoint with the best development change.
  2. Score the held-out eval on it:
  3. Deploy it: