> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trymonte.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Scaling and Versions

> Scale deployments and roll out new checkpoints

A deployment runs on replicas: one model server each, on one or more GPUs. Monte adds and removes replicas between a minimum and a maximum. Each change of checkpoint makes a new version behind the same name and endpoint.

## Choose How to Scale

| Mode | Description |
| - | - |
| On traffic | Adds replicas when requests wait, and removes them when traffic drops |
| On a schedule | Keeps a set minimum in stated hours, and `--min-replicas` outside them |
| To zero | Removes all replicas after a period with no requests |

The modes combine: a schedule sets the minimum, and traffic adds replicas above it.

```bash theme={null}
# On traffic
monte deployment update support-triage \
  --min-replicas 1 --max-replicas 6 --target-concurrency 32

# On a schedule
monte deployment update support-triage \
  --schedule "Mon-Fri 07:00-19:00 America/New_York min=2" --min-replicas 0

# To zero
monte deployment update support-triage --min-replicas 0 --scale-down-delay 15m
```

Monte adds a replica when the average concurrent requests for each replica go above `--target-concurrency`. It removes one after load stays below the target for `--scale-down-delay`. An adapter deployment scales with its base, and scaling flags refuse with `E_REFUSAL` on an adapter.

Monte replaces a failed replica, and replicas spread across the zones of the region.

## Queueing and Cold Starts

* **Queue.** When all replicas are busy, requests wait, up to `--max-queue` (default 256). A full queue returns `429` with a `Retry-After` header.
* **Timeout.** A request that does not finish in 600 seconds, queue time included, returns `504`.
* **Zero replicas.** The endpoint returns `503` with a `Retry-After` header, and the request starts a replica. That first request waits for a GPU instance and for the weights to load.
* **Billing.** The GPU instance is billed until its last replica stops.
* **Quota.** With [BYOC](https://docs.trymonte.ai/deployment.md), the cloud account's GPU quota caps `--max-replicas`.

## Choose a GPU Type

```bash theme={null}
monte deployment update support-triage --gpu h100 --gpus-per-replica 1
```

`--gpu` takes `l4`, `l40s`, `a100-80gb`, `h100`, or `h200`. `--gpus-per-replica` takes 1, 2, 4, or 8. A change of GPU type starts new replicas and then stops the old ones.

| Model | Weight size | GPUs per replica |
| - | - | - |
| 7-8B, BF16 | about 16 GB | One L4 with a short context, or one L40S, A100 80GB, or H100 |
| 14B, BF16 | about 28 GB | One A100 80GB or one H100 |
| 27-32B, BF16 | about 64 GB | One H100 with a small KV cache, two GPUs, or FP8 weights |
| Qwen3-30B-A3B, BF16 (MoE) | about 61 GB | One H100 |
| gpt-oss-120b, MXFP4 (MoE) | about 63 GB | One H100 |
| Qwen3-235B-A22B, BF16 (MoE) | about 470 GB | Eight H100, or four H100 with FP8 weights (about 235 GB) |

A mixture-of-experts (MoE) model needs memory for its total parameters, not its active parameters. The KV cache needs more memory on top. A full fine-tune of gpt-oss-120b saves BF16 weights, about 234 GB, and needs four H100 or more.

## Deploy a New Version

Each version points to one checkpoint. The deployment keeps old versions until it is deleted, so a rollback needs no new deploy.

| Version | Checkpoint | Traffic |
| - | - | - |
| `v2` | `train-11/300` | 10% |
| `v1` | `train-07/200` | 90% |

**1. Start the New Version**

```bash theme={null}
monte deployment update support-triage --checkpoint support-triage/train-11 --traffic 10
```

Monte moves the share when the new version is `ready`. The split is per request, and each response carries a `Monte-Version` header. Without `--traffic`, `--checkpoint` moves all traffic.

**2. Watch the New Version**

```bash theme={null}
monte deployment show support-triage --metrics
```

The output prints requests, error rates, latency, and tokens per second for each version. The **Inference** view in the Console shows the same metrics.

**3. Raise the Share**

```bash theme={null}
monte deployment update support-triage --traffic 50
monte deployment update support-triage --traffic 100
```

With no `--checkpoint`, `--traffic` sets the share of the newest version. At 100, the previous version stops its replicas and stays in the version list.

## Roll Back

```bash theme={null}
monte deployment rollback support-triage
monte deployment rollback support-triage --to v1
```

Rollback sends all traffic to the previous version at once, or to the version that `--to` names. If that version has no replicas, the endpoint returns `503` with a `Retry-After` header, and the request starts a replica.

## Pause or Delete a Deployment

```bash theme={null}
monte deployment update support-triage --min-replicas 0 --max-replicas 0
monte deployment delete support-triage
```

A paused deployment (`--min-replicas 0 --max-replicas 0`) keeps its versions, and requests return `503`.

> [!WARNING]
>
> Delete removes all versions. To stop traffic but keep the versions, pause the deployment.

After a delete, requests to the name return `404`. The checkpoints and the runs stay. `monte checkpoint delete` refuses with `E_IN_USE` while a deployment version points to the checkpoint.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.