> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trymonte.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Experiments, evals, baselines, and regrades

These commands fix the evals that runs are compared on, and score models on them. [Experiments](https://docs.trymonte.ai/evaluation/experiments.md) explains held-out and development evals. [Scoring Models](https://docs.trymonte.ai/evaluation/scoring-models.md) explains the baseline and the change.

| Command | Description |
| - | - |
| [`monte experiment create`](#monte-experiment-create) | Create an experiment |
| [`monte experiment list`](#monte-experiment-list) | List experiments |
| [`monte experiment show`](#monte-experiment-show) | Show the evals, baselines, and newest differences of an experiment |
| [`monte experiment baselines`](#monte-experiment-baselines) | Show which evals have a baseline |
| [`monte experiment delete`](#monte-experiment-delete) | Remove an experiment |
| [`monte experiment sync`](#monte-experiment-sync) | Copy run history from artifact storage |
| [`monte experiment push`](#monte-experiment-push) | Copy run history to artifact storage |
| [`monte eval`](#monte-eval) | Score a model or a checkpoint on one eval |
| [`monte grade`](#monte-grade) | Score the replies of an eval run again with another grader |

## monte experiment create

```text theme={null}
monte experiment create <name> --held-out <evals> [--dev <evals>] [--description <text>] [--json]
```

Create an experiment. The command pins each eval by fingerprint with its own eval settings. Nothing in the experiment changes after that.

| Flag | Type | Default | Description |
| - | - | - | - |
| `<name>` | argument | required | Lowercase letters, digits, and `-` |
| `--held-out` | list | required | The held-out evals, comma-separated. Each is an eval environment, as `<name>` or `<name>@<version>`. |
| `--dev` | list | none | The development evals, comma-separated |
| `--description` | string | none | Free text for the record |
| `--json` | flag | off | Print JSON on stdout |

```bash theme={null}
monte experiment create support-triage --held-out support-escalation --dev support-escalation-dev \
  --description "Escalate or resolve support tickets"
```

```text theme={null}
experiment 'support-triage' created (1 held-out, 1 dev)
  held-out support-escalation@1      fingerprint 3f9a1c0e7b2d4a61
  dev      support-escalation-dev@1  fingerprint a04e9d2c5f1b8e37
```

| Error | Cause |
| - | - |
| `E_EXPERIMENT_EXISTS` | An experiment has that name |
| `E_NOT_FOUND` | A name is not an eval environment in artifact storage |
| `E_REFUSAL` | A name is a training environment, or a held-out eval overlaps the task rows of a training environment |

## monte experiment list

```text theme={null}
monte experiment list [--remote] [--json]
```

List experiments with owner, creation time, eval count, and run count.

| Flag | Type | Default | Description |
| - | - | - | - |
| `--remote` | flag | off | Read artifact storage instead of this machine |
| `--json` | flag | off | Print JSON on stdout |

```bash theme={null}
monte experiment list
```

```text theme={null}
NAME                OWNER             CREATED      EVALS               RUNS
support-triage      lee@example.com   2026-10-14   1 held-out, 1 dev   9
```

## monte experiment show

```text theme={null}
monte experiment show <name> [--json]
```

Show the evals of an experiment, its base models, and one line for each eval with a baseline: the baseline score, the newest score, and the paired mean difference of task scores. [Read the Result](https://docs.trymonte.ai/evaluation/scoring-models.md#read-the-result) explains the line.

| Flag | Type | Default | Description |
| - | - | - | - |
| `<name>` | argument | required | The experiment |
| `--json` | flag | off | Print JSON on stdout |

```bash theme={null}
monte experiment show support-triage
```

```text theme={null}
evals       held-out support-escalation@1    dev support-escalation-dev@1
models      Qwen/Qwen3-8B @ a09a3545

  support-escalation-dev: 0.50 -> 0.60 (delta +0.10 ± 0.06 paired, 30 fixed / 10 broken, eval@200)
  support-escalation:     0.49 -> 0.58 (delta +0.09 ± 0.05 paired, 24 fixed / 6 broken, eval@200)
```

## monte experiment baselines

```text theme={null}
monte experiment baselines <name> [--json]
```

Show which evals have a baseline for each base model. `monte train` refuses until every eval has one.

| Flag | Type | Default | Description |
| - | - | - | - |
| `<name>` | argument | required | The experiment |
| `--json` | flag | off | Print JSON on stdout |

```bash theme={null}
monte experiment baselines support-triage
```

```text theme={null}
model Qwen/Qwen3-8B @ a09a3545
  support-escalation-dev  dev        eval-01  0.50
  support-escalation      held-out   eval-02  0.49
ready to train
```

## monte experiment delete

```text theme={null}
monte experiment delete <name> --confirm <name> [--json]
```

Remove an experiment, its run history on this machine and in artifact storage, and the weights of its checkpoints. Environment and dataset versions stay in artifact storage.

> [!WARNING]
>
> A delete cannot be undone.

| Flag | Type | Default | Description |
| - | - | - | - |
| `<name>` | argument | required | The experiment |
| `--confirm` | string | required | The experiment name, typed in full |
| `--json` | flag | off | Print JSON on stdout |

```bash theme={null}
monte experiment delete support-triage-v2 --confirm support-triage-v2
```

```text theme={null}
Deleted experiment support-triage-v2: 3 runs, 0 checkpoints.
```

| Error | Cause |
| - | - |
| `E_IN_USE` | A deployment serves a checkpoint of the experiment |

## monte experiment sync

```text theme={null}
monte experiment sync [<name>...] [--json]
```

Copy run history from artifact storage to this machine. It never deletes a local file.

| Flag | Type | Default | Description |
| - | - | - | - |
| `<name>...` | arguments | all experiments | The experiments to copy |
| `--json` | flag | off | Print JSON on stdout |

```bash theme={null}
monte experiment sync support-triage
```

```text theme={null}
support-triage  4 new runs, 2 updated, 812 files (38 MB)
```

## monte experiment push

```text theme={null}
monte experiment push <name> [--run <run>] [--json]
```

Copy the run history of one experiment to artifact storage, which the Console reads.

| Flag | Type | Default | Description |
| - | - | - | - |
| `<name>` | argument | required | The experiment |
| `--run` | string | all runs | Push only one run |
| `--json` | flag | off | Print JSON on stdout |

```bash theme={null}
monte experiment push support-triage --run eval-03
```

```text theme={null}
support-triage/eval-03  pushed 204 files (9.1 MB)
```

## monte eval

```text theme={null}
monte eval <experiment> [<eval>] [--model <hf-id>] [--checkpoint <run>/<step> | --step <n>]
  [--gpu <type>] [--gpus <n>] [--remote | --push] [--smoke] [--dry-run] [--yes] [--json]
```

Score one model on one eval of an experiment, and record an eval run. With no `--model`, `--checkpoint`, or `--step`, the target is the newest checkpoint. The eval settings come from the experiment, and no flag changes them.

| Flag | Type | Default | Description |
| - | - | - | - |
| `<experiment>` | argument | required | The experiment |
| `<eval>` | argument | the only development eval | The eval to score. A held-out eval is never the default. |
| `--model` | string | none | Score this base model, as a Hugging Face model ID pinned by its snapshot revision. Required when the experiment has no checkpoint. |
| `--checkpoint` | string | the newest checkpoint | One exact checkpoint, as `<run>/<step>` |
| `--step` | integer | | The checkpoint at one absolute step. Refuses when two runs saved a checkpoint at that step. |
| `--gpu` | string | the smallest type that fits the model | `l4`, `l40s`, `a100-80gb`, `h100`, or `h200`. With `--remote` only. |
| `--gpus` | integer | the smallest count that fits the model | The number of GPUs, with `--remote` only. A count above 8 spans several nodes. |
| `--remote` | flag | off | Submit the eval to a GPU instance and return |
| `--push` | flag | off | Run here and upload the results after each state change |
| `--smoke` | flag | off | Score the first 3 tasks. A smoke run never counts as a result or a baseline. |
| `--dry-run` | flag | off | Print the target, the eval, and the settings, and start nothing |
| `--yes` | flag | off | Skip the `launch?` prompt |
| `--json` | flag | off | Print JSON on stdout |

`--model`, `--checkpoint`, and `--step` exclude each other. `--remote` does not combine with `--push` or `--dry-run`.

```bash theme={null}
monte eval support-triage --checkpoint train-07/200 --remote
```

```text theme={null}
run        support-triage/eval-03 (eval)
eval       support-escalation-dev@1 (dev, 200 tasks)
target     checkpoint train-07/200 on Qwen/Qwen3-8B @ a09a3545
gpus       h100 x1
launch? [y/N] y
Submitted support-triage/eval-03 to gpu-4f2a. Follow it with: monte run logs support-triage/eval-03 -f
```

| Error | Cause |
| - | - |
| `E_REFUSAL` | No `<eval>` and more than one development eval, no `--model` and no checkpoint, or `--step` matches two checkpoints |
| `E_PREFLIGHT` | The eval changed its fingerprint, or the run lock is held |
| `E_NOT_FOUND` | No such experiment, eval, or checkpoint |

## monte grade

```text theme={null}
monte grade <experiment>/<run> --into <experiment> [--eval <eval>]
  [--remote | --push] [--dry-run] [--yes] [--json]
```

Score the replies of an eval run again with the grader of another experiment's eval, and record a new eval run there. No model is served: the command reads the original run's [traces](https://docs.trymonte.ai/observability/traces.md). The new run counts as a result in the target experiment, and a regraded baseline is a baseline there. [Fix a Grader and Regrade](https://docs.trymonte.ai/environments/graders.md#fix-a-grader-and-regrade) shows the full sequence.

| Flag | Type | Default | Description |
| - | - | - | - |
| `<experiment>/<run>` | argument | required | The original eval run |
| `--into` | string | required | The target experiment. It cannot be the experiment of the original run. |
| `--eval` | string | the one eval whose tasks match | The eval in the target experiment, when more than one has the same tasks |
| `--remote` | flag | off | Run the grader on a GPU instance and return, for a judge model that needs a GPU |
| `--push` | flag | off | Run here and upload the results after each state change |
| `--dry-run` | flag | off | Print the original run, the target eval, and the task match, and start nothing |
| `--yes` | flag | off | Skip the `launch?` prompt |
| `--json` | flag | off | Print JSON on stdout |

```bash theme={null}
monte grade support-triage/eval-03 --into support-triage-v2
```

```text theme={null}
tasks      200 of 200 match by task hash  settings match
launch? [y/N] y
support-triage-v2/eval-03 succeeded  support-escalation-dev=0.57±0.07  (200 tasks, 0 served)
```

| Error | Cause |
| - | - |
| `E_NOT_COMPARABLE` | The target eval has other tasks, or other eval settings, than the original eval |
| `E_REFUSAL` | The original run is not an eval run, is a smoke run, or has no stored traces |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.