Evaluation

An evaluation runs a model on a set of tasks, scores what comes back, and reports how the model performed. The result tells you what is working, what is not, and where the next effort should go. When you fine-tune a model on new data, or add a tool to its agent harness, you expect it to do the work better. The change on its own does not confirm that. An evaluation is how you find out objectively: it puts a number on the model’s performance, so you can say whether the change measurably improved it and by how much. That only holds if the two numbers you compare were produced the same way. If the tasks, the sampling rules, or the grader moved between them, the difference could have come from any of those rather than from the model, and the comparison no longer says anything about the change you made. A standardized evaluation removes that doubt, which is why every evaluation in Monte runs against a fixed Measurement.

​

Model capability or agent capability

A score covers the model and the agent harness together. The harness picks the tools, holds the prompt, and decides when a task is done. Each of those moves the number. To read one of them, hold the other still.

• Hold the harness, change the model. The score reads model capability.
• Hold the model, change the harness. The score reads harness quality. A new tool can raise accuracy and triple the token cost at the same time.

A Measurement freezes the harness in its fingerprint. Two Runs under one Measurement used the same harness, so the difference between their scores belongs to the model. Changing the harness changes the fingerprint, which makes a different Measurement. Monte does not score across Measurements, so it does not compare harnesses today.

​

What makes an evaluation good

Coverage. The tasks have to look like the work you care about. A narrow set of tasks gives a high score that says little about real use. The grader. The grader defines what good means for the Environment. A noisy or biased grader scores the wrong thing, and training then optimizes for it. Programmatic checks such as an exact match, a test suite, or code that runs are steadier than a model that judges a model. Representative tasks. Public benchmarks let you compare against published results. Your own tasks tell you whether the model works for your use. Most useful Environments have both. Repeats. One attempt per task is noisy. More attempts separate a real gain from variance. The Measurement fixes the number of attempts in its eval settings, so it cannot drift between Runs. Reproducibility. The same input has to return the same score. The fingerprint hashes the task files, the grader, the prompt, and the harness, and Monte refuses a Run when any of them moved.

​

Related pages

​

Further reading

• Evaluation (NeMo Gym)
• LLM evaluation: Why testing AI models matters (IBM)