In This Section
One Round
-
Collect. A deployment keeps each request and its response as a trace. Collection is on by default, and the traces stay in artifact storage, which is in the team’s own account with BYOC.
--traces offon a deployment keeps none. -
Review. Download recent traces and pick the ones worth learning from: wrong answers, hand-offs that a person overturned, and new kinds of requests.
A trace records what the model did, not whether it was right. The app can tag each request with a
metadatafield, such as a ticket ID, so the review joins traces with the outcome in the team’s own ticketing system. -
Turn them into data. Each trace is a chat
messagesrow, the same format as an SFT dataset.- A corrected reply becomes a demonstration row in a new dataset version.
- A reviewer’s edit and the model’s reply become a preference pair.
- A new kind of request becomes a task in the training environment.
-
Retrain. Start a run from the deployed checkpoint,
train-11/300, with the same method that made it. The new dataset version holds the earlier rows and the new ones, so the model keeps what it learned. - Ship. Score the new checkpoint on the development eval, and compare it task by task with the deployed checkpoint. Ship it only if it fixes more tasks than it breaks, then move traffic to it in steps.
Privacy
--anonymize replaces personal details in the dataset rows before they are stored. The traces themselves keep the original text in artifact storage, and --traces off keeps none. Anonymization lists what it finds. A team’s traces train only that team’s models.
