> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trymonte.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# DPO

> Train on preferred and rejected responses

Direct preference optimization makes the preferred response of each pair more likely than the rejected one. It fits when reviewers can pick the better of two outputs but cannot easily write the best one.

## Prepare the Data

DPO trains on a [dataset](https://docs.trymonte.ai/datasets/overview.md) added with `--kind preference`. Each row of `support-prefs` holds a ticket, a preferred reply, and a rejected reply:

```json theme={null}
{"prompt": "How do I reset my password? The reset link never arrives.", "chosen": "Sorry about that. The link goes to the email on your account, so check your spam folder first. Then request a new link. It expires in 1 hour.", "rejected": "Password reset emails are sent to the registered address. Verify the spam folder. Request a new link if necessary. Links expire after 1 hour."}
```

```bash theme={null}
monte dataset add support-prefs --from-file reviews.jsonl --kind preference
```

Both replies are correct, and the pair teaches the tone. A reviewer's edit of a model reply also gives a pair: the edit is `chosen`, and the model reply is `rejected`.

## Start a Run

DPO usually starts from an [SFT](https://docs.trymonte.ai/training/sft.md) checkpoint. That checkpoint is the reference model.

```bash theme={null}
monte train support-triage \
  --config triage-dpo \
  --data support-prefs \
  --steps 100 \
  --parent train-07/200
```

## Parameters

| Parameter | Description |
| - | - |
| `dpo.beta` | How far the model can move from the reference. A lower value allows a larger change. |
| `batch_size` | Pairs in one step |
| `learning_rate` | The peak learning rate |
| `max_seq_length` | The prompt plus the longer response, in tokens |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.