DPO - Monte Platform

Direct preference optimization makes the preferred response of each pair more likely than the rejected one. It fits when reviewers can pick the better of two outputs but cannot easily write the best one.

Prepare the Data

DPO trains on a dataset added with --kind preference. Each row of support-prefs holds a ticket, a preferred reply, and a rejected reply:

{"prompt": "How do I reset my password? The reset link never arrives.", "chosen": "Sorry about that. The link goes to the email on your account, so check your spam folder first. Then request a new link. It expires in 1 hour.", "rejected": "Password reset emails are sent to the registered address. Verify the spam folder. Request a new link if necessary. Links expire after 1 hour."}

monte dataset add support-prefs --from-file reviews.jsonl --kind preference

Both replies are correct, and the pair teaches the tone. A reviewer’s edit of a model reply also gives a pair: the edit is chosen, and the model reply is rejected.

Start a Run

DPO usually starts from an SFT checkpoint. That checkpoint is the reference model.

monte train support-triage \
  --config triage-dpo \
  --data support-prefs \
  --steps 100 \
  --parent train-07/200

Parameters

Parameter	Description
`dpo.beta`	How far the model can move from the reference. A lower value allows a larger change.
`batch_size`	Pairs in one step
`learning_rate`	The peak learning rate
`max_seq_length`	The prompt plus the longer response, in tokens