Skip to main content
An experiment is a fixed set of eval environments with pinned settings. Every run in it is scored on the same tasks under the same rules, so two scores differ only because the model differs.

​
Eval Roles

Choose the checkpoint on the development evals, then score the held-out eval on that checkpoint only. An experiment can have any number of development evals.

​
Create an Experiment

Both flags take a comma-separated list of eval environments (Environments). --held-out is required. Monte refuses a held-out eval whose tasks overlap the task rows of a training environment. The experiment pins each eval by its fingerprint, the eval settings, and the roles. Monte checks every fingerprint before each run and refuses the run if an eval changed. A changed eval needs a new experiment.

​
Take the Baselines

Score the base model on every eval before the first training run:
Every later score is a change from this baseline. monte train refuses to start until each eval has a baseline for its base model. A second base model needs its own baselines.