<checkpoint> | argument | none | <experiment>/<run>/<step>, or <experiment>/<run> for the last checkpoint of the run |
--model | string | none | A Hugging Face model ID, deployed as a base deployment with no training |
--name | string | required | The deployment name. Requests send it as model. |
--base | string | none | A running base deployment to load the adapter into |
--cloud | string | the only connected cloud | The cloud that runs the deployment |
--gpu | string | the smallest GPU type that fits the model and its context | l4, l40s, a100-80gb, h100, or h200 |
--gpus-per-replica | integer | the smallest count that fits the model and its context | GPUs for each replica: 1, 2, 4, 8, or 16. 16 spans two nodes. |
--context | integer | the model’s maximum | The longest context in tokens. A shorter context needs less KV cache memory. |
--quantization | string | none | none, or fp8 to serve FP8 weights at about half the memory |
--min-replicas | integer | 1 | The lowest replica count. 0 allows scale to zero. |
--max-replicas | integer | 1 | The highest replica count |
--target-concurrency | integer | 32 | Concurrent requests for each replica. Above it, Monte adds replicas. |
--schedule | string | none | A window with its own minimum, such as "Mon-Fri 07:00-19:00 America/New_York min=2". Repeat it to add windows. |
--scale-down-delay | duration | 10m | The time below the target before Monte removes a replica, and the idle time before the last one stops |
--max-queue | integer | 256 | Requests that wait across all replicas of the deployment before the endpoint returns 429 |
--max-adapters | integer | 8 | With --model only. Adapters in GPU memory at once. |
--max-adapter-rank | integer | 64 | With --model only. The highest LoRA rank that the base deployment accepts. |
--traces | string | on | Keep each request and response as a trace in artifact storage, or off to keep no content |
--private | flag | off | Serve on a private IP in the team’s network. BYOC only. |
--json | flag | off | Print JSON on stdout |