Serving

A deployment serves one model behind an OpenAI-compatible endpoint: a checkpoint, a base model, or a LoRA adapter on a base deployment. With BYOC, it runs on GPU instances in the team’s account. A deployment runs, and is billed, until it is deleted or scaled to zero.

Choose a Checkpoint

Deploy only a checkpoint that beat the baseline on the held-out evals. monte checkpoint show prints those scores.

Deploy a Fine-Tuned Model

monte deployment create support-triage/train-07 --name support-triage \
  --cloud gcp --gpu h100 --min-replicas 1 --max-replicas 4

support-triage/train-07 names the checkpoint that the run saved at its last step, train-07/200. --name is also the model value in each request. Without --gpu, Monte picks the smallest GPU type that fits the model and its context. The command checks each servable file against its sha256: the safetensors weights, or the PEFT adapter of a LoRA run. If a file fails, the command refuses with E_VERIFICATION. Upload the weights again with monte checkpoint push, then run the command again. The training config decides whether a checkpoint holds full weights or a LoRA adapter.

	Full model	LoRA adapter
What the checkpoint holds	All weights of the model	Low-rank weights on top of a base model
GPU use	One server for each deployment	Many adapters share one base deployment

Without --base, an adapter gets its own base-model server that other adapters cannot share. Callers need an API key.

Deploy a Base Model

--model deploys a Hugging Face model with no training. Each create takes exactly one of <checkpoint> or --model.

monte deployment create --model Qwen/Qwen3-8B --name qwen3-8b \
  --max-adapters 8 --max-adapter-rank 64

Add Adapters to a Base

--base deploys a LoRA checkpoint onto a running base deployment.

monte deployment create support-triage/train-11 --name support-triage-dpo --base qwen3-8b

Each adapter is its own deployment with its own name, and requests for different adapters share the base model’s GPUs. monte key create --deployment support-triage-dpo limits a key to one adapter.

--max-adapters sets how many adapters stay in GPU memory at once (default 8). Other adapters load from CPU memory when a request needs them.
--max-adapter-rank sets the highest adapter rank that the base accepts (default 64). An adapter’s LoRA rank must not exceed it, and a higher maximum takes more GPU memory.
An adapter deployment has no replicas of its own. Its requests count toward the base deployment’s scaling, and scaling flags refuse with E_REFUSAL on an adapter.
An adapter works only on the base model that it was trained on.

Check Status

monte deployment show support-triage

Status	Description
`pending`	Monte waits for a GPU instance
`loading`	The server loads the weights
`ready`	The endpoint accepts requests
`updating`	A new version or GPU type starts. The current version serves requests.
`idle`	Zero replicas. The next request starts one.
`paused`	Zero replicas and a maximum of zero. Requests return `503`.
`failed`	The server did not start. `monte deployment show` prints the reason.

monte deployment list prints every deployment. monte deployment logs support-triage -f follows the model server’s logs.

Run Outside Monte

monte checkpoint download writes the servable weights of a checkpoint in the Hugging Face layout. A LoRA run gives a PEFT adapter that vLLM can load.

monte checkpoint download support-triage/train-07/200 --dest ./models/train-07
vllm serve Qwen/Qwen3-8B --enable-lora --max-lora-rank 64 \
  --lora-modules support-triage=./models/train-07

Get Started

Training

Inference

Datasets

Environments

Evaluation

Observability

Security

Choose a Checkpoint

Deploy a Fine-Tuned Model

Deploy a Base Model

Add Adapters to a Base

Check Status

Run Outside Monte

​Choose a Checkpoint

​Deploy a Fine-Tuned Model

​Deploy a Base Model

​Add Adapters to a Base

​Check Status

​Run Outside Monte

Choose a Checkpoint

Deploy a Fine-Tuned Model

Deploy a Base Model

Add Adapters to a Base

Check Status

Run Outside Monte