Skip to main content
A Measurement is a frozen evaluation contract over exactly one Environment. It fixes what you measure and how you measure it. An Experiment uses one Measurement by default but may use more.

​
Definition

The Environment supplies the graded tasks. The Measurement freezes how evaluation uses them: which tasks, with which eval settings, against which grader. A Measurement holds no model or training configuration. Each Run binds its model and exact inputs. A training Run records its training data, recipe, and hyperparameters. Runs under the same Measurement can compare different approaches under the same evaluation contract.

​
What gets frozen

monte init freezes three things:
  1. Evaluation Splits: the task partitions used for evaluation, with each file’s path, sha256, and task count pinned in measurement.json.
  2. Eval settings: the sampling and grading settings that every eval Run uses.
  3. Fingerprint: a content hash of the task source, the grader, and the harness config.
The freeze is what makes scores comparable. Two Runs under one Measurement answer the same tasks under the same rules. A changed fingerprint is a different Measurement, not a new version of the same one.
Monte recomputes the fingerprint before each Run. If the Environment changed since monte init, Monte refuses the Run.

​
Lifecycle

monte init <measurement> --env <name> creates the Measurement. Every later command names it, as in monte eval math or monte status math. Each eval or train Run appends one row to the Measurement’s Ledger. A Lineage and its Baseline belong to one Measurement and never cross it.

Environment

The graded task package a Measurement freezes.

The loop

How Splits, Baselines, and deltas judge improvement.

Runs, lineages, and the ledger

The record every Run writes to.