Skip to main content
Every round of the loop adds a new version of the training data. The deployed model then improves round by round, while each round stays comparable with earlier rounds in the same experiment.

Traces Into Datasets

Reviewed traces go into a new version of a dataset. A corrected reply is a demonstration row for SFT, and a reviewer’s edit with the model’s original reply is a pair for DPO. Add the earlier rows and the new ones together, and the dataset gets a new version, such as support-replies@2. Each run records the exact version it trained on.

Traces Into Environments

A trace’s request becomes a task row in the training environment. The task takes the trace’s messages and tools in the task format. For a conversation over several requests, join its traces by a session ID in metadata. The reviewer fills in the grader fields, such as should_escalate and expected_category. Adding the task file again makes a new version, such as support-escalation-train@2, for a GRPO run:

Each Round

Train from the deployed checkpoint with --parent, on the new versions. Pick the checkpoint on the development eval, and compare it task by task with the deployed checkpoint. Score the held-out eval once for each round. Its change from the baseline compares directly with every earlier round, but a held-out eval scored over many rounds starts to overstate gains, so refresh it with new traffic in a new experiment every few rounds. Then move traffic in steps, and the deployment keeps collecting traces for the next round. Every step is a CLI command, so a team can run a round from a script on its own schedule.

New Kinds of Requests

When traffic shifts, pinned evals no longer cover it. Add the new requests to an eval environment as a new version, and create a new experiment with it. Take its baselines before the next run. Each monte dataset add and monte env add --training checks the new rows against every eval, so a trace that matches an eval task does not leak into training unnoticed.