The loop

The loop is the platform’s reason to exist. One turn: train one chunk, eval the new Checkpoint on dev, read the paired delta against the Baseline. Repeat until you stop. Then make one claim on test.

​

Baseline first

Before any training, monte eval scores the base model on the dev Split, and once on test. That is eval@0, the Baseline. You read every later score in the Lineage as a paired delta against it. A number with no Baseline is not evidence, and monte train refuses a real chunk without one (E_NO_BASELINE). Only a smoke Run skips this gate.

​

What counts as evidence

Only admissible Runs count. A --smoke run is a capped sanity pass: never a Baseline, never a chain parent, never an eval target, never a promote source. monte status marks such rows [not evidence].

​

Cycles and strikes

A Cycle is the first admissible eval at a new highest step in its Lineage. A retry at the same step is not a new Cycle. Each Cycle gets a strike verdict: strike means the paired dev delta is not significantly positive. Strikes are labels, nothing more. Nothing stops the loop automatically. The only stop is yours.

​

Paired deltas

Two Runs under one Measurement answered the same tasks. So Monte pairs the results per Task and tests the delta directly, McNemar-style. This is usually, not always, tighter than comparing two independent score intervals. When per-task files are unreachable, monte status falls back to a labelled unpaired interval. A task-set mismatch does not degrade: it refuses (E_CORRUPT_STATE).

​

Dev decides, test never does

The dev Split drives every decision: the eval curve, strike verdicts, checkpoint selection, your stop. The test Split is touched exactly twice: at Baseline and at the final claim. Monte refuses a third admissible test eval. It also refuses a test eval at a step the tool did not select.

​

The final claim

When you stop the loop, the tool selects the Checkpoint with the highest paired dev delta. You eval that Checkpoint on test. That result is the claim. Dev selects, test confirms. One grader does all of it: training, eval, and spot-checks use the same grader, so no agreement gap can open.

​

Promote or honest null

If the claim shows improvement, monte promote exports the Checkpoint’s servable folder, checksum-verified, behind a typed confirm. If it does not, that is the honest null: nothing is promoted, the Ledger keeps the record, and the result still counts as an answer.

​

Related pages