> ## Documentation Index > Fetch the complete documentation index at: https://docs.trymonte.ai/llms.txt > Use this file to discover all available pages before exploring further. # Training > Training runs and checkpoints These commands train a model and manage the checkpoints that the runs save. [Training Overview](https://docs.trymonte.ai/training/overview.md) explains how to choose a method. | Command | Description | | - | - | | [`monte train`](#monte-train) | Start a training run | | [`monte train resume`](#monte-train-resume) | Continue a run for more steps, as a new run | | [`monte checkpoint list`](#monte-checkpoint-list) | List the checkpoints of an experiment | | [`monte checkpoint show`](#monte-checkpoint-show) | Show one checkpoint | | [`monte checkpoint push`](#monte-checkpoint-push) | Upload weights to artifact storage | | [`monte checkpoint download`](#monte-checkpoint-download) | Download weights from artifact storage | | [`monte checkpoint delete`](#monte-checkpoint-delete) | Remove weights | ## monte train ```text theme={null} monte train --config --data --steps (--from-base | --parent /) [--model ] [--gpu ] [--gpus ] [--remote | --push] [--dry-run] [--yes] [--json] monte train --config --data --smoke [--model ] [--gpu ] [--gpus ] [--remote | --push] [--dry-run] [--yes] [--json] ``` Start a training run in an experiment. The run saves a checkpoint every `checkpoint_every` steps of the [training config](https://docs.trymonte.ai/training/configs.md) (default 50) and at its last step. Steps count from the base model. | Flag | Type | Default | Description | | - | - | - | - | | `` | argument | required | The experiment that records the run | | `--config` | string | required | A [training config](https://docs.trymonte.ai/training/configs.md): a name from a config directory, or a path to a YAML file | | `--data` | string | required | The training data: a training environment or a dataset, as `` or `@` | | `--steps` | integer | required | The number of trainer steps in this run. A run from `train-07/200` with `--steps 100` ends at step 300. | | `--from-base` | flag | | Start from the base model at step 0 | | `--parent` | string | | Start from one checkpoint, as `/`. State `--from-base` or `--parent`, never both. | | `--model` | string | the config's `model` | A Hugging Face model ID that replaces the base model for this run | | `--gpu` | string | the smallest type that fits the model and method | `l4`, `l40s`, `a100-80gb`, `h100`, or `h200`. With `--remote` only. | | `--gpus` | integer | the smallest count that fits the model and method | The number of GPUs, with `--remote` only. A count above 8 spans several nodes. | | `--remote` | flag | off | Submit the run to a GPU instance and return. The CLI uploads the config with the job. | | `--push` | flag | off | Upload results after each state change, and push the checkpoint to artifact storage at the end | | `--smoke` | flag | off | A 3-step training run that never counts as a result. It takes no `--steps`, `--from-base`, or `--parent`. | | `--dry-run` | flag | off | Print the resolved run settings and the full trainer config, and start nothing | | `--yes` | flag | off | Skip the `launch?` prompt | | `--json` | flag | off | Print JSON on stdout | ```bash theme={null} monte train support-triage --config triage-sft --data support-replies --steps 200 \ --from-base --model Qwen/Qwen3-8B ``` ```text theme={null} run support-triage/train-07 (train) data support-replies@1 (dataset, sft, 1,240 rows) start base Qwen/Qwen3-8B @ a09a3545 steps 0 -> 200 gpus h100 x1 launch? [y/N] y succeeded support-triage/train-07 checkpoint train-07/200 took 41m ``` | Error | Cause | | - | - | | `E_NO_BASELINE` | An eval in the experiment has no baseline for this base model | | `E_REFUSAL` | No starting point, both starting points, a config that sets a key the run owns, or training data that the method does not take | | `E_PREFLIGHT` | The run lock is held, an eval changed its fingerprint, the disk is full, or another process uses the GPU | ## monte train resume ```text theme={null} monte train resume / --steps [--remote | --push] [--dry-run] [--yes] [--json] ``` Continue a succeeded training run for more steps, as a new run. The new run takes the parent's training config, training data, weights, optimizer state, and data position, so the curve continues from the last step. | Flag | Type | Default | Description | | - | - | - | - | | `/` | argument | required | The succeeded training run to continue | | `--steps` | integer | required | The number of additional trainer steps | | `--remote` | flag | off | Submit the run to a GPU instance and return | | `--push` | flag | off | Upload results after each state change, and push the checkpoint at the end | | `--dry-run` | flag | off | Print the resolved run settings, and start nothing | | `--yes` | flag | off | Skip the `launch?` prompt | | `--json` | flag | off | Print JSON on stdout | ```bash theme={null} monte train resume support-triage/train-07 --steps 100 --remote ``` ```text theme={null} run support-triage/train-09 (train), resumes train-07 start train-07/200, with optimizer state steps 200 -> 300 launch? [y/N] y Submitted support-triage/train-09 to gpu-4f2a. ``` To change the config or the data, use `monte train --parent` instead. That run takes the weights only. | Error | Cause | | - | - | | `E_REFUSAL` | The run did not succeed, or its config file changed its hash since the run | | `E_NOT_FOUND` | The run's resume weights are not on this machine and not in artifact storage | ## monte checkpoint list ```text theme={null} monte checkpoint list [--run ] [--json] ``` List the checkpoints of an experiment, newest first. | Flag | Type | Default | Description | | - | - | - | - | | `` | argument | required | The experiment | | `--run` | string | all runs | Only the checkpoints of one run | | `--json` | flag | off | Print JSON on stdout | ```bash theme={null} monte checkpoint list support-triage ``` ```text theme={null} CHECKPOINT CONFIG DATA PARENT WEIGHTS train-11/300 triage-dpo support-prefs train-07/200 local, stored (both) train-09/250 triage-sft support-replies train-07/200 local (both) train-07/200 triage-sft support-replies base local, stored (both) ... ``` `local` means the weights are on this machine, and `stored` means they are in artifact storage. The kind in brackets is `servable`, `resume`, or `both`. ## monte checkpoint show ```text theme={null} monte checkpoint show [--json] ``` Show one checkpoint in full: the run, the base model, the files and their sizes, the eval scores, and the deployments that serve it. | Flag | Type | Default | Description | | - | - | - | - | | `` | argument | required | `//`, or `/` for the last checkpoint of the run | | `--json` | flag | off | Print JSON on stdout | ```bash theme={null} monte checkpoint show support-triage/train-07/200 ``` ```text theme={null} checkpoint support-triage/train-07/200 (LoRA adapter, rank 64) base Qwen/Qwen3-8B @ a09a3545 servable stored 612 MB scores support-escalation-dev 0.60 (delta +0.10 ± 0.06 paired, eval-03) support-escalation 0.58 (delta +0.09 ± 0.05 paired, eval-05) deployments support-triage (v1) ``` ## monte checkpoint push ```text theme={null} monte checkpoint push [--kind servable|resume|both] [--json] ``` Upload the weights of a checkpoint from this machine to artifact storage, so any GPU instance can evaluate, resume, or deploy it. `--push` on `monte train` does the same at the end of a run. | Flag | Type | Default | Description | | - | - | - | - | | `` | argument | required | `//`, or `/` | | `--kind` | string | `both` | `servable` is the Hugging Face folder that a model server loads. `resume` is the full set, with optimizer state, that training continues from. | | `--json` | flag | off | Print JSON on stdout | ```bash theme={null} monte checkpoint push support-triage/train-09/250 ``` ```text theme={null} Pushed support-triage/train-09/250 to gs://helpdesk-monte-artifacts. ``` ## monte checkpoint download ```text theme={null} monte checkpoint download [--kind servable|resume] [--dest ] [--force] [--json] ``` Download the weights of a checkpoint from artifact storage. With no `--dest`, the files go under the data root, where `monte eval` and `monte train` find them. A servable folder loads with `AutoModelForCausalLM.from_pretrained`, or with PEFT for an adapter. | Flag | Type | Default | Description | | - | - | - | - | | `` | argument | required | `//`, or `/` | | `--kind` | string | `servable` | `servable` or `resume` | | `--dest` | path | the recorded path | A directory for the files | | `--force` | flag | off | Download again when the folder is already complete | | `--json` | flag | off | Print JSON on stdout | ```bash theme={null} monte checkpoint download support-triage/train-07/200 --dest ./models/train-07 ``` ```text theme={null} Wrote ./models/train-07 (adapter for Qwen/Qwen3-8B @ a09a3545). ``` | Error | Cause | | - | - | | `E_NOT_FOUND` | No weights of that kind are in artifact storage. Push them first with `monte checkpoint push`. | ## monte checkpoint delete ```text theme={null} monte checkpoint delete --kind servable|resume|all [--yes] [--json] ``` Remove the weights of a checkpoint from artifact storage and from the GPU instances. The run and its scores stay in run history. > [!WARNING] > > A delete with `--kind servable` or `all` cannot be undone. The checkpoint can no longer be evaluated or deployed. | Flag | Type | Default | Description | | - | - | - | - | | `` | argument | required | `//`, or `/` | | `--kind` | string | required | `resume` frees the optimizer state and keeps the servable weights. `servable` and `all` remove more. | | `--yes` | flag | off | Skip the prompt | | `--json` | flag | off | Print JSON on stdout | ```bash theme={null} monte checkpoint delete support-triage/train-09/250 --kind resume --yes ``` ```text theme={null} Deleted resume weights of support-triage/train-09/250 (1.9 GB) from artifact storage and gpu-4f2a. ``` | Error | Cause | | - | - | | `E_IN_USE` | A version of a deployment uses the servable weights. Old versions count until the deployment is deleted. | This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.