Skip to main content
A deployed model meets requests that no eval covered. Continual learning turns that traffic into the next round of training. A deployment records each request as a trace. The team reviews the traces, turns the useful ones into dataset rows or tasks, trains from the deployed checkpoint, and ships the result behind the same endpoint.

In This Section

One Round

  1. Collect. A deployment keeps each request and its response as a trace. Collection is on by default, and the traces stay in artifact storage, which is in the team’s own account with BYOC. --traces off on a deployment keeps none.
  2. Review. Download recent traces and pick the ones worth learning from: wrong answers, hand-offs that a person overturned, and new kinds of requests.
    A trace records what the model did, not whether it was right. The app can tag each request with a metadata field, such as a ticket ID, so the review joins traces with the outcome in the team’s own ticketing system.
  3. Turn them into data. Each trace is a chat messages row, the same format as an SFT dataset.
    • A corrected reply becomes a demonstration row in a new dataset version.
    • A reviewer’s edit and the model’s reply become a preference pair.
    • A new kind of request becomes a task in the training environment.
  4. Retrain. Start a run from the deployed checkpoint, train-11/300, with the same method that made it. The new dataset version holds the earlier rows and the new ones, so the model keeps what it learned.
  5. Ship. Score the new checkpoint on the development eval, and compare it task by task with the deployed checkpoint. Ship it only if it fixes more tasks than it breaks, then move traffic to it in steps.
Each step is a CLI command, so a team runs a round on its own schedule, or Monte’s engineers run it with the team. An experiment’s evals never change, so new tasks in an eval need a new experiment. Training data can change in every round.

Privacy

--anonymize replaces personal details in the dataset rows before they are stored. The traces themselves keep the original text in artifact storage, and --traces off keeps none. Anonymization lists what it finds. A team’s traces train only that team’s models.