Training

These commands train a model and manage the checkpoints that the runs save. Training Overview explains how to choose a method.

Command	Description
`monte train`	Start a training run
`monte train resume`	Continue a run for more steps, as a new run
`monte checkpoint list`	List the checkpoints of an experiment
`monte checkpoint show`	Show one checkpoint
`monte checkpoint push`	Upload weights to artifact storage
`monte checkpoint download`	Download weights from artifact storage
`monte checkpoint delete`	Remove weights

monte train

monte train <experiment> --config <name|path> --data <name> --steps <n>
  (--from-base | --parent <run>/<step>) [--model <hf-id>] [--gpu <type>] [--gpus <n>]
  [--remote | --push] [--dry-run] [--yes] [--json]
monte train <experiment> --config <name|path> --data <name> --smoke
  [--model <hf-id>] [--gpu <type>] [--gpus <n>] [--remote | --push] [--dry-run] [--yes] [--json]

Start a training run in an experiment. The run saves a checkpoint every checkpoint_every steps of the training config (default 50) and at its last step. Steps count from the base model.

Flag	Type	Default	Description
`<experiment>`	argument	required	The experiment that records the run
`--config`	string	required	A training config: a name from a config directory, or a path to a YAML file
`--data`	string	required	The training data: a training environment or a dataset, as `<name>` or `<name>@<version>`
`--steps`	integer	required	The number of trainer steps in this run. A run from `train-07/200` with `--steps 100` ends at step 300.
`--from-base`	flag		Start from the base model at step 0
`--parent`	string		Start from one checkpoint, as `<run>/<step>`. State `--from-base` or `--parent`, never both.
`--model`	string	the config’s `model`	A Hugging Face model ID that replaces the base model for this run
`--gpu`	string	the smallest type that fits the model and method	`l4`, `l40s`, `a100-80gb`, `h100`, or `h200`. With `--remote` only.
`--gpus`	integer	the smallest count that fits the model and method	The number of GPUs, with `--remote` only. A count above 8 spans several nodes.
`--remote`	flag	off	Submit the run to a GPU instance and return. The CLI uploads the config with the job.
`--push`	flag	off	Upload results after each state change, and push the checkpoint to artifact storage at the end
`--smoke`	flag	off	A 3-step training run that never counts as a result. It takes no `--steps`, `--from-base`, or `--parent`.
`--dry-run`	flag	off	Print the resolved run settings and the full trainer config, and start nothing
`--yes`	flag	off	Skip the `launch?` prompt
`--json`	flag	off	Print JSON on stdout

monte train support-triage --config triage-sft --data support-replies --steps 200 \
  --from-base --model Qwen/Qwen3-8B

run        support-triage/train-07 (train)
data       support-replies@1 (dataset, sft, 1,240 rows)
start      base Qwen/Qwen3-8B @ a09a3545
steps      0 -> 200
gpus       h100 x1
launch? [y/N] y
succeeded  support-triage/train-07  checkpoint train-07/200  took 41m

Error	Cause
`E_NO_BASELINE`	An eval in the experiment has no baseline for this base model
`E_REFUSAL`	No starting point, both starting points, a config that sets a key the run owns, or training data that the method does not take
`E_PREFLIGHT`	The run lock is held, an eval changed its fingerprint, the disk is full, or another process uses the GPU

monte train resume

monte train resume <experiment>/<run> --steps <n>
  [--remote | --push] [--dry-run] [--yes] [--json]

Continue a succeeded training run for more steps, as a new run. The new run takes the parent’s training config, training data, weights, optimizer state, and data position, so the curve continues from the last step.

Flag	Type	Default	Description
`<experiment>/<run>`	argument	required	The succeeded training run to continue
`--steps`	integer	required	The number of additional trainer steps
`--remote`	flag	off	Submit the run to a GPU instance and return
`--push`	flag	off	Upload results after each state change, and push the checkpoint at the end
`--dry-run`	flag	off	Print the resolved run settings, and start nothing
`--yes`	flag	off	Skip the `launch?` prompt
`--json`	flag	off	Print JSON on stdout

monte train resume support-triage/train-07 --steps 100 --remote

run        support-triage/train-09 (train), resumes train-07
start      train-07/200, with optimizer state
steps      200 -> 300
launch? [y/N] y
Submitted support-triage/train-09 to gpu-4f2a.

To change the config or the data, use monte train --parent instead. That run takes the weights only.

Error	Cause
`E_REFUSAL`	The run did not succeed, or its config file changed its hash since the run
`E_NOT_FOUND`	The run’s resume weights are not on this machine and not in artifact storage

monte checkpoint list

monte checkpoint list <experiment> [--run <run>] [--json]

List the checkpoints of an experiment, newest first.

Flag	Type	Default	Description
`<experiment>`	argument	required	The experiment
`--run`	string	all runs	Only the checkpoints of one run
`--json`	flag	off	Print JSON on stdout

monte checkpoint list support-triage

CHECKPOINT     CONFIG        DATA                       PARENT         WEIGHTS
train-11/300   triage-dpo    support-prefs              train-07/200   local, stored (both)
train-09/250   triage-sft    support-replies            train-07/200   local (both)
train-07/200   triage-sft    support-replies            base           local, stored (both)
...

local means the weights are on this machine, and stored means they are in artifact storage. The kind in brackets is servable, resume, or both.

monte checkpoint show

monte checkpoint show <checkpoint> [--json]

Show one checkpoint in full: the run, the base model, the files and their sizes, the eval scores, and the deployments that serve it.

Flag	Type	Default	Description
`<checkpoint>`	argument	required	`<experiment>/<run>/<step>`, or `<experiment>/<run>` for the last checkpoint of the run
`--json`	flag	off	Print JSON on stdout

monte checkpoint show support-triage/train-07/200

checkpoint  support-triage/train-07/200 (LoRA adapter, rank 64)
base        Qwen/Qwen3-8B @ a09a3545
servable    stored  612 MB
scores      support-escalation-dev  0.60 (delta +0.10 ± 0.06 paired, eval-03)
            support-escalation      0.58 (delta +0.09 ± 0.05 paired, eval-05)
deployments support-triage (v1)

monte checkpoint push

monte checkpoint push <checkpoint> [--kind servable|resume|both] [--json]

Upload the weights of a checkpoint from this machine to artifact storage, so any GPU instance can evaluate, resume, or deploy it. --push on monte train does the same at the end of a run.

Flag	Type	Default	Description
`<checkpoint>`	argument	required	`<experiment>/<run>/<step>`, or `<experiment>/<run>`
`--kind`	string	`both`	`servable` is the Hugging Face folder that a model server loads. `resume` is the full set, with optimizer state, that training continues from.
`--json`	flag	off	Print JSON on stdout

monte checkpoint push support-triage/train-09/250

Pushed support-triage/train-09/250 to gs://helpdesk-monte-artifacts.

monte checkpoint download

monte checkpoint download <checkpoint> [--kind servable|resume] [--dest <dir>] [--force] [--json]

Download the weights of a checkpoint from artifact storage. With no --dest, the files go under the data root, where monte eval and monte train find them. A servable folder loads with AutoModelForCausalLM.from_pretrained, or with PEFT for an adapter.

Flag	Type	Default	Description
`<checkpoint>`	argument	required	`<experiment>/<run>/<step>`, or `<experiment>/<run>`
`--kind`	string	`servable`	`servable` or `resume`
`--dest`	path	the recorded path	A directory for the files
`--force`	flag	off	Download again when the folder is already complete
`--json`	flag	off	Print JSON on stdout

monte checkpoint download support-triage/train-07/200 --dest ./models/train-07

Wrote ./models/train-07 (adapter for Qwen/Qwen3-8B @ a09a3545).

Error	Cause
`E_NOT_FOUND`	No weights of that kind are in artifact storage. Push them first with `monte checkpoint push`.

monte checkpoint delete

monte checkpoint delete <checkpoint> --kind servable|resume|all [--yes] [--json]

Remove the weights of a checkpoint from artifact storage and from the GPU instances. The run and its scores stay in run history.

A delete with --kind servable or all cannot be undone. The checkpoint can no longer be evaluated or deployed.

Flag	Type	Default	Description
`<checkpoint>`	argument	required	`<experiment>/<run>/<step>`, or `<experiment>/<run>`
`--kind`	string	required	`resume` frees the optimizer state and keeps the servable weights. `servable` and `all` remove more.
`--yes`	flag	off	Skip the prompt
`--json`	flag	off	Print JSON on stdout

monte checkpoint delete support-triage/train-09/250 --kind resume --yes

Deleted resume weights of support-triage/train-09/250 (1.9 GB) from artifact storage and gpu-4f2a.

Error	Cause
`E_IN_USE`	A version of a deployment uses the servable weights. Old versions count until the deployment is deleted.

Commands

MCP

monte train

monte train resume

monte checkpoint list

monte checkpoint show

monte checkpoint push

monte checkpoint download

monte checkpoint delete

​monte train

​monte train resume

​monte checkpoint list

​monte checkpoint show

​monte checkpoint push

​monte checkpoint download

​monte checkpoint delete

monte train

monte train resume

monte checkpoint list

monte checkpoint show

monte checkpoint push

monte checkpoint download

monte checkpoint delete