Experiments

An Experiment is a fixed standard for scoring a model. Two Runs under the same Experiment answered the same tasks under the same rules, so you can compare them directly. An Experiment is built on one Environment, and it fixes which of that Environment’s tasks run, the sampling rules, and what counts as correct.

​

Why an Experiment exists

Evaluation has many variables you can set. You can give each Task one attempt or several, change the decoding temperature, run a different subset of the tasks, or mark answers more or less strictly. Each of those returns a different number for the same model, and none of them is wrong. They answer different questions about how well the model does the task. An Experiment holds those variables constant, so within one Experiment the model is the only thing that changed between two Runs. You can define as many Experiments as you want. When the variables go unrecorded, old results stop being usable. You cannot tell why two numbers differ, so you cannot compare the experiments that produced them.

​

What an Experiment fixes

An Experiment fixes three things when you create it.

1. The tasks. Which tasks run, as an exact list. Not a count and not a name, but those specific tasks.
2. The eval settings. How the model is sampled: the decoding, the temperature, and how many attempts each Task gets.
3. The grading. Which grader scores the answers, and the prompt the model saw.

An Experiment fixes the evaluation only, not the training. The Recipe stays free, and so does the Source, what a Run trains on, because those are what the driving agent changes between Runs. An Experiment names no model either: each Run binds its own, so many models can compete under one Experiment. Monte checks the frozen material before every Run. If any of it changed, that is a different Experiment, and Monte refuses to score the new Run against the old ones.

​

The Splits

An Experiment owns its Splits, and the trainer and the evaluator both read the same lists, which is what holds the roles below in place. An Experiment over a trainable Environment carries all three. One over an eval-only Environment carries dev alone when that Environment was frozen from a single benchmark, and dev and test when it was frozen from two, which is the form that can carry a final claim.

• train. What the model learns on.
• dev. What the driving agent reads to choose its next Move.
• test. Held back for the Baseline and the final claim.

The driving agent picks each next Move from what dev shows, so the loop fits itself to those tasks over time. A dev score therefore overstates the model, which is why dev is not the number you report. test stays out of the loop entirely, and it gets two reads: you take the first one yourself, as a deliberate eval of the base model before any training begins, and the second is the final claim, after which Monte refuses a third. Skip that first read and Monte refuses the claim as well, because there would be nothing to compare it against. Improvement covers how those reads happen.

​

Related pages