Start a training run in an experiment. The run saves a checkpoint every checkpoint_every steps of the recipe (default 50) and at its last step. Steps count from the base model.
Flag
Type
Default
Description
<experiment>
argument
required
The experiment that records the run
--recipe
string
required
A recipe: a name from a recipe directory, or a path to a YAML file
--data
string
required
The training data: a training environment or a dataset, as <name> or <name>@<version>
--steps
integer
required
The number of trainer steps in this run. A run from train-07/200 with --steps 100 ends at step 300.
--from-base
flag
Start from the base model at step 0
--parent
string
Start from one checkpoint, as <run>/<step>. State --from-base or --parent, never both.
--model
string
the recipe’s model
A Hugging Face model ID that replaces the base model for this run
--gpu
string
the smallest type that fits the model and method
l4, l40s, a100-80gb, h100, or h200. With --remote only.
--gpus
integer
the smallest count that fits the model and method
The number of GPUs, with --remote only. A count above 8 spans several nodes.
--remote
flag
off
Submit the run to a GPU instance and return. The CLI uploads the recipe with the job.
--push
flag
off
Upload results after each state change, and push the checkpoint to artifact storage at the end
--smoke
flag
off
A 3-step training run that never counts as a result. It takes no --steps, --from-base, or --parent.
--dry-run
flag
off
Print the resolved run settings and the full trainer config, and start nothing
run support-triage/train-07 (train)data support-replies@1 (dataset, sft, 6,480 rows)start base Qwen/Qwen3-8B @ a09a3545steps 0 -> 200gpus h100 x1launch? [y/N] ySubmitted support-triage/train-07 to gpu-4f2a. Follow it with: monte run logs support-triage/train-07 -f
Error
Cause
E_NO_BASELINE
An eval in the experiment has no baseline for this base model
E_REFUSAL
No starting point, both starting points, a recipe that sets a key the run owns, or training data that the method does not take
E_PREFLIGHT
The run lock is held, an eval changed its fingerprint, the disk is full, or another process uses the GPU
Add steps to a finished training run, recorded as a new run that points to the original. The new run takes the parent’s recipe, training data, weights, optimizer state, and data position, so the curve continues from the last step.
Flag
Type
Default
Description
<experiment>/<run>
argument
required
The succeeded training run to extend
--steps
integer
required
The number of additional trainer steps
--remote
flag
off
Submit the run to a GPU instance and return
--push
flag
off
Upload results after each state change, and push the checkpoint at the end
--dry-run
flag
off
Print the resolved run settings, and start nothing
--yes
flag
off
Skip the launch? prompt
--json
flag
off
Print JSON on stdout
monte train extend support-triage/train-07 --steps 100 --remote
run support-triage/train-09 (train), extends train-07start train-07/200, with optimizer statesteps 200 -> 300launch? [y/N] ySubmitted support-triage/train-09 to gpu-4f2a. Follow it with: monte run logs support-triage/train-09 -f
To change the recipe or the data, use monte train --parent instead. That run takes the weights only.
Error
Cause
E_REFUSAL
The run did not succeed, or its recipe changed its hash since the run
E_NOT_FOUND
The run’s resume weights are not on this machine and not in artifact storage
monte checkpoint list
monte checkpoint list <experiment> [--run <run>] [--json]
List the checkpoints of an experiment, newest first.
Flag
Type
Default
Description
<experiment>
argument
required
The experiment
--run
string
all runs
Only the checkpoints of one run
--json
flag
off
Print JSON on stdout
monte checkpoint list support-triage
CHECKPOINT RECIPE DATA PARENT WEIGHTStrain-11/300 triage-dpo support-prefs train-07/200 local, stored (both)train-09/250 triage-sft support-replies train-07/200 local (both)train-07/200 triage-sft support-replies base local, stored (both)...
local means the weights are on this machine, and stored means they are in artifact storage. The kind in brackets is servable, resume, or both.
monte checkpoint show
monte checkpoint show <checkpoint> [--json]
Show one checkpoint in full: the run, the base model, the files and their sizes, the eval scores, and the deployments that serve it.
Flag
Type
Default
Description
<checkpoint>
argument
required
<experiment>/<run>/<step>, or <experiment>/<run> for the last checkpoint of the run
monte checkpoint push <checkpoint> [--kind servable|resume|both] [--json]
Upload the weights of a checkpoint from this machine to artifact storage, so any GPU instance can evaluate, resume, or deploy it. --push on monte train does the same at the end of a run.
Flag
Type
Default
Description
<checkpoint>
argument
required
<experiment>/<run>/<step>, or <experiment>/<run>
--kind
string
both
servable is the Hugging Face folder that a model server loads. resume is the full set, with optimizer state, that training continues from.
--json
flag
off
Print JSON on stdout
monte checkpoint push support-triage/train-09/250
Pushed support-triage/train-09/250 to gs://helpdesk-monte-artifacts.
monte checkpoint download
monte checkpoint download <checkpoint> [--kind servable|resume] [--dest <dir>] [--force] [--json]
Download the weights of a checkpoint from artifact storage. With no --dest, the files go under the data root, where monte eval and monte train find them. A servable folder loads with AutoModelForCausalLM.from_pretrained, or with PEFT for an adapter.
Flag
Type
Default
Description
<checkpoint>
argument
required
<experiment>/<run>/<step>, or <experiment>/<run>
--kind
string
servable
servable or resume
--dest
path
the recorded path
A directory for the files
--force
flag
off
Download again when the folder is already complete
--json
flag
off
Print JSON on stdout
monte checkpoint download support-triage/train-07/200 --dest ./models/train-07
Wrote ./models/train-07 (adapter for Qwen/Qwen3-8B @ a09a3545).
Error
Cause
E_NOT_FOUND
No weights of that kind are in artifact storage. Push them first with monte checkpoint push.
monte checkpoint delete
monte checkpoint delete <checkpoint> --kind servable|resume|all [--yes] [--json]
Remove the weights of a checkpoint from artifact storage and from the GPU instances. The run and its scores stay in run history.
A delete with --kind servable or all cannot be undone. The checkpoint can no longer be evaluated or deployed.
Flag
Type
Default
Description
<checkpoint>
argument
required
<experiment>/<run>/<step>, or <experiment>/<run>
--kind
string
required
resume frees the optimizer state and keeps the servable weights. servable and all remove more.
--yes
flag
off
Skip the prompt
--json
flag
off
Print JSON on stdout
monte checkpoint delete support-triage/train-09/250 --kind resume --yes
Deleted resume weights of support-triage/train-09/250 (1.9 GB) from artifact storage and gpu-4f2a.
Error
Cause
E_IN_USE
A version of a deployment uses the servable weights. Old versions count until the deployment is deleted.
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"Monte Platform","colors":{"primary":"#2F6E92","light":"#70A6C1","dark":"#1E3A68"},"logo":{"light":"https://mintcdn.com/monte-inc/gHJ25yqL727CkHQ3/logo/light.png?fit=max\u0026auto=format\u0026n=gHJ25yqL727CkHQ3\u0026q=85\u0026s=70592fed17125283509e24b0eb6d8ffc","dark":"https://mintcdn.com/monte-inc/gHJ25yqL727CkHQ3/logo/dark.png?fit=max\u0026auto=format\u0026n=gHJ25yqL727CkHQ3\u0026q=85\u0026s=dfa5df62f81de461aaad809c789297d1"},"favicon":"https://mintcdn.com/monte-inc/gDE0uEcoDx7_moLr/favicon.png?fit=max\u0026auto=format\u0026n=gDE0uEcoDx7_moLr\u0026q=85\u0026s=1d58f7325198c466585e820513d03b27","contextual":{"options":["copy","view","chatgpt","claude","perplexity","mcp","cursor","vscode"],"display":"header"},"styling":{"latex":false},"navigation":{"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"monte-inc","hasAudioNarration":false,"actualSubdomain":"monte-inc","customDomains":["docs.trymonte.ai"],"gitSource":{"type":"github","owner":"monte-inc","repo":"monte-platform-docs","deployBranch":"main","contentDirectory":"","isPrivate":true,"mountPath":"","label":""},"entitlements":{"AI_CHAT":{"status":"DISABLED"}},"buildId":"6af43f73bf7f51e9bf6b569c","clientVersion":"0.0.4142","searchFilterCounts":[{"language":"default","tag":"Platform","count":31},{"language":"default","tag":"CLI","count":8},{"language":"default","tag":"Command Reference","count":7},{"language":"default","tag":"Training","count":7},{"language":"default","tag":"Deployment","count":5},{"language":"default","tag":"Get Started","count":4},{"language":"default","tag":"Environments","count":4},{"language":"default","tag":"Evaluation","count":4},{"language":"default","tag":"Methods","count":4},{"language":"default","tag":"Continual Learning","count":3}]},"assistantConfig":{"enableAskAiButton":false,"enableStarterQuestions":false,"enableContextualStarterQuestions":false,"enableAskAiSearchBar":true},"basePath":"","hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[],"googleFontsInlined":false},"customCss":[{"filePath":"style.css","content":".mermaid,[class*=\"mermaid\"]{text-align:center}.mermaid>svg,[class*=\"mermaid\"]>svg{display:block;margin-inline:auto}.mermaid .cluster-label text,.mermaid .cluster-label span,.mermaid .cluster-label p,.mermaid .cluster text{fill:#17181B;color:#17181B}.dark .mermaid .cluster-label text,.dark .mermaid .cluster-label span,.dark .mermaid .cluster-label p,.dark .mermaid .cluster text{fill:#F0F1F2;color:#F0F1F2}"}],"page":{"pagePath":"commands/training","slug":"/commands/training","pageMetadata":{"title":"Training and Checkpoints","description":"Training runs and checkpoints","href":"/commands/training"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Training runs and checkpoints\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,code:`code`,p:`p`,pre:`pre`,span:`span`,tbody:`tbody`,td:`td`,th:`th`,thead:`thead`,tr:`tr`,..._provideComponents(),...props.components},{CodeBlock,Heading,Table,Warning}=_components;if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!Table)_missingMdxReference(`Table`,true);if(!Warning)_missingMdxReference(`Warning`,true);return _jsxs(_Fragment,{children:[_jsxs(_components.p,{children:[`These commands train a model and manage the checkpoints that the runs save. `,_jsx(_components.a,{href:`/training/overview`,children:`Training Overview`}),` compares the four methods.`]}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Command`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.a,{href:`#monte-train`,children:_jsx(_components.code,{children:`monte train`})})}),_jsx(_components.td,{children:`Start a training run`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.a,{href:`#monte-train-extend`,children:_jsx(_components.code,{children:`monte train extend`})})}),_jsx(_components.td,{children:`Add steps to a finished run, recorded as a new run that points to the original`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.a,{href:`#monte-checkpoint-list`,children:_jsx(_components.code,{children:`monte checkpoint list`})})}),_jsx(_components.td,{children:`List the checkpoints of an experiment`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.a,{href:`#monte-checkpoint-show`,children:_jsx(_components.code,{children:`monte checkpoint show`})})}),_jsx(_components.td,{children:`Show one checkpoint`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.a,{href:`#monte-checkpoint-push`,children:_jsx(_components.code,{children:`monte checkpoint push`})})}),_jsx(_components.td,{children:`Upload weights to artifact storage`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.a,{href:`#monte-checkpoint-download`,children:_jsx(_components.code,{children:`monte checkpoint download`})})}),_jsx(_components.td,{children:`Download weights from artifact storage`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.a,{href:`#monte-checkpoint-delete`,children:_jsx(_components.code,{children:`monte checkpoint delete`})})}),_jsx(_components.td,{children:`Remove weights`})]})]})]}),`\n`,_jsx(Heading,{level:`2`,id:`monte-train`,children:`monte train`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`6`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`6`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`monte train \u003cexperiment> --recipe \u003cname|path> --data \u003cname> --steps \u003cn>`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:` (--from-base | --parent \u003crun>/\u003cstep>) [--model \u003chf-id>] [--gpu \u003ctype>] [--gpus \u003cn>]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:` [--remote | --push] [--dry-run] [--yes] [--json]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`monte train \u003cexperiment> --recipe \u003cname|path> --data \u003cname> --smoke`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:` [--model \u003chf-id>] [--gpu \u003ctype>] [--gpus \u003cn>] [--remote | --push] [--dry-run] [--yes] [--json]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsxs(_components.p,{children:[`Start a training run in an experiment. The run saves a checkpoint every `,_jsx(_components.code,{children:`checkpoint_every`}),` steps of the `,_jsx(_components.a,{href:`/training/recipes`,children:`recipe`}),` (default 50) and at its last step. Steps count from the base model.`]}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Flag`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Default`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`\u003cexperiment>`})}),_jsx(_components.td,{children:`argument`}),_jsx(_components.td,{children:`required`}),_jsx(_components.td,{children:`The experiment that records the run`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--recipe`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`required`}),_jsxs(_components.td,{children:[`A `,_jsx(_components.a,{href:`/training/recipes`,children:`recipe`}),`: a name from a recipe directory, or a path to a YAML file`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--data`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`required`}),_jsxs(_components.td,{children:[`The training data: a training environment or a dataset, as `,_jsx(_components.code,{children:`\u003cname>`}),` or `,_jsx(_components.code,{children:`\u003cname>@\u003cversion>`})]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--steps`})}),_jsx(_components.td,{children:`integer`}),_jsx(_components.td,{children:`required`}),_jsxs(_components.td,{children:[`The number of trainer steps in this run. A run from `,_jsx(_components.code,{children:`train-07/200`}),` with `,_jsx(_components.code,{children:`--steps 100`}),` ends at step 300.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--from-base`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{}),_jsx(_components.td,{children:`Start from the base model at step 0`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--parent`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{}),_jsxs(_components.td,{children:[`Start from one checkpoint, as `,_jsx(_components.code,{children:`\u003crun>/\u003cstep>`}),`. State `,_jsx(_components.code,{children:`--from-base`}),` or `,_jsx(_components.code,{children:`--parent`}),`, never both.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--model`})}),_jsx(_components.td,{children:`string`}),_jsxs(_components.td,{children:[`the recipe’s `,_jsx(_components.code,{children:`model`})]}),_jsx(_components.td,{children:`A Hugging Face model ID that replaces the base model for this run`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--gpu`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`the smallest type that fits the model and method`}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`l4`}),`, `,_jsx(_components.code,{children:`l40s`}),`, `,_jsx(_components.code,{children:`a100-80gb`}),`, `,_jsx(_components.code,{children:`h100`}),`, or `,_jsx(_components.code,{children:`h200`}),`. With `,_jsx(_components.code,{children:`--remote`}),` only.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--gpus`})}),_jsx(_components.td,{children:`integer`}),_jsx(_components.td,{children:`the smallest count that fits the model and method`}),_jsxs(_components.td,{children:[`The number of GPUs, with `,_jsx(_components.code,{children:`--remote`}),` only. A count above 8 spans several nodes.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--remote`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Submit the run to a GPU instance and return. The CLI uploads the recipe with the job.`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--push`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Upload results after each state change, and push the checkpoint to artifact storage at the end`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--smoke`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsxs(_components.td,{children:[`A 3-step training run that never counts as a result. It takes no `,_jsx(_components.code,{children:`--steps`}),`, `,_jsx(_components.code,{children:`--from-base`}),`, or `,_jsx(_components.code,{children:`--parent`}),`.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--dry-run`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Print the resolved run settings and the full trainer config, and start nothing`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--yes`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsxs(_components.td,{children:[`Skip the `,_jsx(_components.code,{children:`launch?`}),` prompt`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--json`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Print JSON on stdout`})]})]})]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`2`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`monte`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` train`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` support-triage`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --recipe`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` triage-sft`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --data`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` support-replies`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --steps`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 200`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --from-base`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --model`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Qwen/Qwen3-8B`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --remote`})]}),`\n`]})})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`8`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`8`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`run support-triage/train-07 (train)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`data support-replies@1 (dataset, sft, 6,480 rows)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`start base Qwen/Qwen3-8B @ a09a3545`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`steps 0 -> 200`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`gpus h100 x1`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`launch? [y/N] y`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`Submitted support-triage/train-07 to gpu-4f2a. Follow it with: monte run logs support-triage/train-07 -f`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Error`}),_jsx(_components.th,{children:`Cause`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`E_NO_BASELINE`})}),_jsx(_components.td,{children:`An eval in the experiment has no baseline for this base model`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`E_REFUSAL`})}),_jsx(_components.td,{children:`No starting point, both starting points, a recipe that sets a key the run owns, or training data that the method does not take`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`E_PREFLIGHT`})}),_jsx(_components.td,{children:`The run lock is held, an eval changed its fingerprint, the disk is full, or another process uses the GPU`})]})]})]}),`\n`,_jsx(Heading,{level:`2`,id:`monte-train-extend`,children:`monte train extend`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`3`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`3`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`monte train extend \u003cexperiment>/\u003crun> --steps \u003cn>`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:` [--remote | --push] [--dry-run] [--yes] [--json]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsx(_components.p,{children:`Add steps to a finished training run, recorded as a new run that points to the original. The new run takes the parent’s recipe, training data, weights, optimizer state, and data position, so the curve continues from the last step.`}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Flag`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Default`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`\u003cexperiment>/\u003crun>`})}),_jsx(_components.td,{children:`argument`}),_jsx(_components.td,{children:`required`}),_jsx(_components.td,{children:`The succeeded training run to extend`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--steps`})}),_jsx(_components.td,{children:`integer`}),_jsx(_components.td,{children:`required`}),_jsx(_components.td,{children:`The number of additional trainer steps`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--remote`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Submit the run to a GPU instance and return`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--push`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Upload results after each state change, and push the checkpoint at the end`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--dry-run`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Print the resolved run settings, and start nothing`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--yes`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsxs(_components.td,{children:[`Skip the `,_jsx(_components.code,{children:`launch?`}),` prompt`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--json`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Print JSON on stdout`})]})]})]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`monte`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` train`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` extend`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` support-triage/train-07`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --steps`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 100`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --remote`})]}),`\n`]})})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`6`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`6`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`run support-triage/train-09 (train), extends train-07`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`start train-07/200, with optimizer state`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`steps 200 -> 300`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`launch? [y/N] y`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`Submitted support-triage/train-09 to gpu-4f2a. Follow it with: monte run logs support-triage/train-09 -f`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsxs(_components.p,{children:[`To change the recipe or the data, use `,_jsx(_components.code,{children:`monte train --parent`}),` instead. That run takes the weights only.`]}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Error`}),_jsx(_components.th,{children:`Cause`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`E_REFUSAL`})}),_jsx(_components.td,{children:`The run did not succeed, or its recipe changed its hash since the run`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`E_NOT_FOUND`})}),_jsx(_components.td,{children:`The run’s resume weights are not on this machine and not in artifact storage`})]})]})]}),`\n`,_jsx(Heading,{level:`2`,id:`monte-checkpoint-list`,children:`monte checkpoint list`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`monte checkpoint list \u003cexperiment> [--run \u003crun>] [--json]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsx(_components.p,{children:`List the checkpoints of an experiment, newest first.`}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Flag`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Default`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`\u003cexperiment>`})}),_jsx(_components.td,{children:`argument`}),_jsx(_components.td,{children:`required`}),_jsx(_components.td,{children:`The experiment`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--run`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`all runs`}),_jsx(_components.td,{children:`Only the checkpoints of one run`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--json`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Print JSON on stdout`})]})]})]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`monte`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` checkpoint`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` list`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` support-triage`})]}),`\n`]})})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`6`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`6`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`CHECKPOINT RECIPE DATA PARENT WEIGHTS`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`train-11/300 triage-dpo support-prefs train-07/200 local, stored (both)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`train-09/250 triage-sft support-replies train-07/200 local (both)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`train-07/200 triage-sft support-replies base local, stored (both)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`...`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.code,{children:`local`}),` means the weights are on this machine, and `,_jsx(_components.code,{children:`stored`}),` means they are in artifact storage. The kind in brackets is `,_jsx(_components.code,{children:`servable`}),`, `,_jsx(_components.code,{children:`resume`}),`, or `,_jsx(_components.code,{children:`both`}),`.`]}),`\n`,_jsx(Heading,{level:`2`,id:`monte-checkpoint-show`,children:`monte checkpoint show`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`monte checkpoint show \u003ccheckpoint> [--json]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsx(_components.p,{children:`Show one checkpoint in full: the run, the base model, the files and their sizes, the eval scores, and the deployments that serve it.`}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Flag`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Default`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`\u003ccheckpoint>`})}),_jsx(_components.td,{children:`argument`}),_jsx(_components.td,{children:`required`}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`\u003cexperiment>/\u003crun>/\u003cstep>`}),`, or `,_jsx(_components.code,{children:`\u003cexperiment>/\u003crun>`}),` for the last checkpoint of the run`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--json`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Print JSON on stdout`})]})]})]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`monte`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` checkpoint`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` show`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` support-triage/train-07/200`})]}),`\n`]})})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`7`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`7`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`checkpoint support-triage/train-07/200 (LoRA adapter, rank 64)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`base Qwen/Qwen3-8B @ a09a3545`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`servable stored 350 MB`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`scores support-escalation-dev 0.60 (delta +0.10 ± 0.06 paired, eval-03)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:` support-escalation 0.58 (delta +0.09 ± 0.05 paired, eval-05)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`deployments support-triage (v1)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsx(Heading,{level:`2`,id:`monte-checkpoint-push`,children:`monte checkpoint push`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`monte checkpoint push \u003ccheckpoint> [--kind servable|resume|both] [--json]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsxs(_components.p,{children:[`Upload the weights of a checkpoint from this machine to artifact storage, so any GPU instance can evaluate, resume, or deploy it. `,_jsx(_components.code,{children:`--push`}),` on `,_jsx(_components.code,{children:`monte train`}),` does the same at the end of a run.`]}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Flag`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Default`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`\u003ccheckpoint>`})}),_jsx(_components.td,{children:`argument`}),_jsx(_components.td,{children:`required`}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`\u003cexperiment>/\u003crun>/\u003cstep>`}),`, or `,_jsx(_components.code,{children:`\u003cexperiment>/\u003crun>`})]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--kind`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:_jsx(_components.code,{children:`both`})}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`servable`}),` is the Hugging Face folder that a model server loads. `,_jsx(_components.code,{children:`resume`}),` is the full set, with optimizer state, that training continues from.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--json`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Print JSON on stdout`})]})]})]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`monte`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` checkpoint`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` push`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` support-triage/train-09/250`})]}),`\n`]})})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`Pushed support-triage/train-09/250 to gs://helpdesk-monte-artifacts.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsx(Heading,{level:`2`,id:`monte-checkpoint-download`,children:`monte checkpoint download`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`monte checkpoint download \u003ccheckpoint> [--kind servable|resume] [--dest \u003cdir>] [--force] [--json]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsxs(_components.p,{children:[`Download the weights of a checkpoint from artifact storage. With no `,_jsx(_components.code,{children:`--dest`}),`, the files go under the data root, where `,_jsx(_components.code,{children:`monte eval`}),` and `,_jsx(_components.code,{children:`monte train`}),` find them. A servable folder loads with `,_jsx(_components.code,{children:`AutoModelForCausalLM.from_pretrained`}),`, or with PEFT for an adapter.`]}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Flag`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Default`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`\u003ccheckpoint>`})}),_jsx(_components.td,{children:`argument`}),_jsx(_components.td,{children:`required`}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`\u003cexperiment>/\u003crun>/\u003cstep>`}),`, or `,_jsx(_components.code,{children:`\u003cexperiment>/\u003crun>`})]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--kind`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:_jsx(_components.code,{children:`servable`})}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`servable`}),` or `,_jsx(_components.code,{children:`resume`})]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--dest`})}),_jsx(_components.td,{children:`path`}),_jsx(_components.td,{children:`the recorded path`}),_jsx(_components.td,{children:`A directory for the files`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--force`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Download again when the folder is already complete`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--json`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Print JSON on stdout`})]})]})]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`monte`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` checkpoint`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` download`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` support-triage/train-07/200`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --dest`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ./models/train-07`})]}),`\n`]})})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`Wrote ./models/train-07 (adapter for Qwen/Qwen3-8B @ a09a3545).`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Error`}),_jsx(_components.th,{children:`Cause`})]})}),_jsx(_components.tbody,{children:_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`E_NOT_FOUND`})}),_jsxs(_components.td,{children:[`No weights of that kind are in artifact storage. Push them first with `,_jsx(_components.code,{children:`monte checkpoint push`}),`.`]})]})})]}),`\n`,_jsx(Heading,{level:`2`,id:`monte-checkpoint-delete`,children:`monte checkpoint delete`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`monte checkpoint delete \u003ccheckpoint> --kind servable|resume|all [--yes] [--json]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsx(_components.p,{children:`Remove the weights of a checkpoint from artifact storage and from the GPU instances. The run and its scores stay in run history.`}),`\n`,_jsx(Warning,{children:_jsxs(_components.p,{children:[`A delete with `,_jsx(_components.code,{children:`--kind servable`}),` or `,_jsx(_components.code,{children:`all`}),` cannot be undone. The checkpoint can no longer be evaluated or deployed.`]})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Flag`}),_jsx(_components.th,{children:`Type`}),_jsx(_components.th,{children:`Default`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`\u003ccheckpoint>`})}),_jsx(_components.td,{children:`argument`}),_jsx(_components.td,{children:`required`}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`\u003cexperiment>/\u003crun>/\u003cstep>`}),`, or `,_jsx(_components.code,{children:`\u003cexperiment>/\u003crun>`})]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--kind`})}),_jsx(_components.td,{children:`string`}),_jsx(_components.td,{children:`required`}),_jsxs(_components.td,{children:[_jsx(_components.code,{children:`resume`}),` frees the optimizer state and keeps the servable weights. `,_jsx(_components.code,{children:`servable`}),` and `,_jsx(_components.code,{children:`all`}),` remove more.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--yes`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Skip the prompt`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`--json`})}),_jsx(_components.td,{children:`flag`}),_jsx(_components.td,{children:`off`}),_jsx(_components.td,{children:`Print JSON on stdout`})]})]})]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`monte`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` checkpoint`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` delete`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` support-triage/train-09/250`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --kind`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` resume`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` --yes`})]}),`\n`]})})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`Deleted resume weights of support-triage/train-09/250 (1.9 GB) from artifact storage and gpu-4f2a.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Error`}),_jsx(_components.th,{children:`Cause`})]})}),_jsx(_components.tbody,{children:_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`E_IN_USE`})}),_jsx(_components.td,{children:`A version of a deployment uses the servable weights. Old versions count until the deployment is deleted.`})]})})]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Training and Checkpoints","description":"Training runs and checkpoints","href":"/commands/training"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"monte train","slug":"monte-train","depth":2,"children":[]},{"title":"monte train extend","slug":"monte-train-extend","depth":2,"children":[]},{"title":"monte checkpoint list","slug":"monte-checkpoint-list","depth":2,"children":[]},{"title":"monte checkpoint show","slug":"monte-checkpoint-show","depth":2,"children":[]},{"title":"monte checkpoint push","slug":"monte-checkpoint-push","depth":2,"children":[]},{"title":"monte checkpoint download","slug":"monte-checkpoint-download","depth":2,"children":[]},{"title":"monte checkpoint delete","slug":"monte-checkpoint-delete","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":[],"subdomain":"monte-inc","siteName":"Monte Platform","title":"Training and Checkpoints - Monte Platform","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-11-11T10:12:06.433Z","originalFileLocation":"commands/training.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Training runs and checkpoints\">\u003cmeta data-mint-head name=\"application-name\" content=\"Monte Platform\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/monte-inc/P2VsY79V6xu55Vfe/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/monte-inc/P2VsY79V6xu55Vfe/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/monte-inc/P2VsY79V6xu55Vfe/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/monte-inc/P2VsY79V6xu55Vfe/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/monte-inc/P2VsY79V6xu55Vfe/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/monte-inc/P2VsY79V6xu55Vfe/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/monte-inc/P2VsY79V6xu55Vfe/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/monte-inc/P2VsY79V6xu55Vfe/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/monte-inc/P2VsY79V6xu55Vfe/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Training and Checkpoints - Monte Platform\">\u003cmeta data-mint-head property=\"og:description\" content=\"Training runs and checkpoints\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Monte Platform\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.trymonte.ai/commands/training\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://monte-inc.mintlify.site/_mintlify/api/og/v2?division=Command+Reference\u0026amp;title=Training+and+Checkpoints\u0026amp;description=Training+runs+and+checkpoints\u0026amp;theme=3ad5c8fd614f9d2a5568ef8b\u0026amp;signature=417193c46f5bf6237f42ad2a6fc4b4451da58bd3749db5d0bfa4ba3911ab460a\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Training and Checkpoints - Monte Platform\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Training runs and checkpoints\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://monte-inc.mintlify.site/_mintlify/api/og/v2?division=Command+Reference\u0026amp;title=Training+and+Checkpoints\u0026amp;description=Training+runs+and+checkpoints\u0026amp;theme=3ad5c8fd614f9d2a5568ef8b\u0026amp;signature=417193c46f5bf6237f42ad2a6fc4b4451da58bd3749db5d0bfa4ba3911ab460a\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/monte-inc/P2VsY79V6xu55Vfe/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Monte Platform\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#2F6E92\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.trymonte.ai/commands/training\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/commands/training.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.trymonte.ai/#organization\",\"name\":\"Monte Platform\",\"url\":\"https://docs.trymonte.ai\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/monte-inc/gHJ25yqL727CkHQ3/logo/light.png?fit=max\u0026auto=format\u0026n=gHJ25yqL727CkHQ3\u0026q=85\u0026s=70592fed17125283509e24b0eb6d8ffc\"}},{\"@type\":\"WebSite\",\"@id\":\"https://docs.trymonte.ai#website\",\"name\":\"Monte Platform\",\"url\":\"https://docs.trymonte.ai\",\"publisher\":{\"@id\":\"https://docs.trymonte.ai/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.trymonte.ai/commands/training#webpage\",\"url\":\"https://docs.trymonte.ai/commands/training\",\"name\":\"Training and Checkpoints\",\"description\":\"Training runs and checkpoints\",\"dateModified\":\"2026-11-11T10:12:06.433Z\",\"isPartOf\":{\"@id\":\"https://docs.trymonte.ai#website\"},\"breadcrumb\":{\"@id\":\"https://docs.trymonte.ai/commands/training#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.trymonte.ai/commands/training#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Command Reference\",\"item\":\"https://docs.trymonte.ai/commands/setup\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Training and Checkpoints\",\"item\":\"https://docs.trymonte.ai/commands/training\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://docs.trymonte.ai/commands/training#article\",\"headline\":\"Training and Checkpoints\",\"name\":\"Training and Checkpoints\",\"description\":\"Training runs and checkpoints\",\"url\":\"https://docs.trymonte.ai/commands/training\",\"mainEntityOfPage\":{\"@id\":\"https://docs.trymonte.ai/commands/training#webpage\"},\"dateModified\":\"2026-11-11T10:12:06.433Z\",\"publisher\":{\"@id\":\"https://docs.trymonte.ai/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.trymonte.ai#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Monte Platform\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"Monte Platform\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/commands/training.md","siteName":"Monte Platform","nav":{"global":null,"tabs":[{"tab":"Platform","icon":"layer-group","pages":[{"title":"Platform","href":""}]},{"tab":"CLI","icon":"terminal","groups":[{"group":"Command Reference","pages":[{"title":"Account and Cloud","href":"/commands/setup"},{"title":"Environments","href":"/commands/environments"},{"title":"Datasets","href":"/commands/datasets"},{"title":"Experiments and Evals","description":"Experiments, evals, baselines, and rescores","href":"/commands/evaluations"},{"title":"Training and Checkpoints","href":"/commands/training"},{"title":"Deployments and Keys","description":"Deployments and inference API keys","href":"/commands/serving"},{"title":"Runs and Traces","href":"/commands/observability"}]},{"group":"MCP","pages":[{"title":"MCP Server","tag":"Coming soon","href":"/mcp"}]}]}]},"basePath":"","title":"Training and Checkpoints - Monte Platform","buildId":"dpl_G2Qd45vZZ7MkBiGF9yYx8ZsbbKXX","deploymentId":"dpl_G2Qd45vZZ7MkBiGF9yYx8ZsbbKXX","dynamics":[],"chunks":[]}