Skip to main content
A deployment runs on replicas: one model server each, on one or more GPUs. Monte adds and removes replicas between a minimum and a maximum. Each change of checkpoint makes a new version behind the same name and endpoint.

​
Choose How to Scale

The modes combine: a schedule sets the minimum, and traffic adds replicas above it.
Monte adds a replica when the average concurrent requests for each replica go above --target-concurrency. It removes one after load stays below the target for --scale-down-delay. An adapter deployment scales with its base, and scaling flags refuse with E_REFUSAL on an adapter. Monte replaces a failed replica, and replicas spread across the zones of the region.

​
Queueing and Cold Starts

  • Queue. When all replicas are busy, requests wait, up to --max-queue (default 256). A full queue returns 429 with a Retry-After header.
  • Timeout. A request that does not finish in 600 seconds, queue time included, returns 504.
  • Zero replicas. The endpoint returns 503 with a Retry-After header, and the request starts a replica. That first request waits for a GPU instance and for the weights to load.
  • Billing. The GPU instance is billed until its last replica stops.
  • Quota. With BYOC, the cloud account’s GPU quota caps --max-replicas.

​
Choose a GPU Type

--gpu takes l4, l40s, a100-80gb, h100, or h200. --gpus-per-replica takes 1, 2, 4, or 8. A change of GPU type starts new replicas and then stops the old ones. A mixture-of-experts (MoE) model needs memory for its total parameters, not its active parameters. The KV cache needs more memory on top. A full fine-tune of gpt-oss-120b saves BF16 weights, about 234 GB, and needs four H100 or more.

​
Deploy a New Version

Each version points to one checkpoint. The deployment keeps old versions until it is deleted, so a rollback needs no new deploy.
1

Start the New Version

Monte moves the share when the new version is ready. The split is per request, and each response carries a Monte-Version header. Without --traffic, --checkpoint moves all traffic.
2

Watch the New Version

The output prints requests, error rates, latency, and tokens per second for each version. The Inference view in the Console shows the same metrics.
3

Raise the Share

With no --checkpoint, --traffic sets the share of the newest version. At 100, the previous version stops its replicas and stays in the version list.

​
Roll Back

Rollback sends all traffic to the previous version at once, or to the version that --to names. If that version has no replicas, the endpoint returns 503 with a Retry-After header, and the request starts a replica.

​
Pause or Delete a Deployment

A paused deployment (--min-replicas 0 --max-replicas 0) keeps its versions, and requests return 503.
Delete removes all versions. To stop traffic but keep the versions, pause the deployment.
After a delete, requests to the name return 404. The checkpoints and the runs stay. monte checkpoint delete refuses with E_IN_USE while a deployment version points to the checkpoint.