Direct Preference Optimization

From @rafailovDirectPreferenceOptimization2023.

Direct Prefercen Optimization, or DPO, optimizes for human preferences while avoiding reinforcmente learning steps: existing methods, like Reinforcement Learning with Human Feedback first fit a reward model to a dataset of prompts and humen preferences over pair of responses, then use Reinforcement Learning to find a policy that maximizes the learned reward.

DPO directly optimizes for the policy best satisfying the preference with a simple classification objective, fitting an implicit reward model whose corresponding optimal policy can be extracted in closed form.

The key insight is to leverage an analytical mapping from reward functions to optimal policies, basically transforming a loss function over reward functions into a loss function over policies.
This enables the avoidance of fitting an explicit, standalone reward model, while still optimizing under existing models of human preferences, such as the Bradley-Terry model.

Formally

Starting from the same RL objective as RLHF:

$$\underset{{\pi }_{\theta }}{\max }{\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }(y|x)}[r_{\varphi }(x,y)]-\beta {\mathbb{D}}_{KL}[{\pi }_{\theta }(y|x)||{\pi }_{ref}(y|x)]$$

is demonstrated that the optimal solution to the KL-constrained reward maximization takes the form:

$${\pi }_r(y|x)=\frac{1}{Z(x)}{\pi }_{ref}(y|x)exp(\frac{1}{\beta }r(x,y))$$

Where $Z(x)={\sum }_y{\pi }_{ref}(y|x)exp(\frac{1}{\beta }(x,y))$ is the partition function. Even using Maximum Likelihood Estimation to estimate $r_{\varphi }$ of the ground truth reward function $r^{\ast }$, it still expensive to estimate the partition function $Z(x)$, which makes this representation hard to utilize in practice.
However, we can rearrange the equation to express the reward function in terms of its corresponding optimal policy ${\pi }_r$, the reference policy ${\pi }_{ref}$, and the unknown partition function $Z(\cdot )$. Specifically, the logarithm of both sides is taken and then with some algebra we obtain:

$$r(x,y)=\beta log\frac{{\pi }_r(y|x)}{{\pi }_{ref}(y|x)}+\beta logZ(x)$$

We can apply the reparameterization to the ground-truth reward $r\ast$ and corresponding optimal model $\pi \ast$. Fortunately, the Bradley-Terry depends only on the difference of rewards between two completition.
Substituting the reparameterization in the equation for $r\ast (x,y)$, the partition function cancels and we can express the human preference probability in terms of only the optimal policy $\pi \ast$ and reference policy ${\pi }_{ref}$. Thus, the optima RLHF policy $\pi \ast$ under the Bradley-Terry model satisfies the preference model:

$$p\ast (y_1\succ y_2|x)=\frac{1}{1+exp(\beta log\frac{\pi \ast (y_2|x)}{{\pi }_{r}ef(y_2|x)}-\beta log\frac{\pi \ast (y1|x)}{{\pi }_{ref}(y_1|x)})}$$

While this uses the Bradley-Terry model, similarly the expression is derivable under the more general Plackett-Luce model.

Now, having the probability of human preference data in terms of the optimal policy rather than the reward model, we can formulate a Maximum Likelihood object for a parametrized policy ${\pi }_{\theta }$. Analogous to the reward modeling approach in RLHF, the policy objective becomes:

$${\mathcal{L}}_{DPO}({\pi }_{\theta };{\pi }_{ref})=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}[log\sigma (\beta log\frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{ref}(y_w|x)})-\beta log\frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{ref}(y_l|x)}]$$

This way, by using an alternative parameterization, we fit an implicit reward whose optimal policy is simply ${\pi }_{\theta }$. Moreover, since is equivalent to fitting a reparametrized Bradley-Terry model, it has certain theoretical properties, such as consistencies under suitable assumption of the preference data distribution.

Mechanistic Understanding

The gradient with respect to parameters $\theta$ can be written as:

Intuitively, the gradient of the loss function ${\mathcal{L}}_{DPO}$ increases the likelihood of the preferred completions $y_w$ and deacreases the likelihood of dispreferred completitions $y_l$. Importantly, the examples are weighted y how much higher the implicit reward model ${\hat{r}}_{\theta }$ rates the dispreferred compeltions, scaled by $\beta$ i.e. how incorrectly the implicit reward models orders the completitions, accounting for the strength of the KL constraint.

DPO Pipeline

The general DPO pipeline is as follows:

1. Sample competitions $y_1,y_2\sim {\pi }_{ref}(\cdot |x)$ for every prompt $x$, and label them with human preferences to construct the offline dataset of preferences $\mathcal{D}=\{x^{(i)},y_w^{(i)},y_l^{(i)}{\}}_{i=1}^N$;
2. Optimize the Language model ${\pi }_{\theta }$ to minimize ${\mathcal{L}}_{DPO}$ for the given ${\pi }_{ref}$ and $\mathcal{D}$ and desired $\beta$.

To reuse publicly available datasets, rather than generating samples and gathering human preferences, we initialize ${\pi }_{ref}={\pi }^{SFT}$ since the preference dataset are sampled using ${\pi }^{SFT}$ (${\pi }^{SFT}$ is the fine-tuned Language Model from which the preference labeled generations were extracted from for the construction of publicly available preference datasets).
However, when ${\pi }^{SFT}$ isn’t available, we initialized ${\pi }_{ref}$ by maximizing likelihood of preferred completitions $(x,y_w)$, that is, ${\pi }_{ref}=argma{x}_{\pi }{\mathbb{E}}_{x,y_w\sim \mathcal{D}}[log\pi (y_w|x)]$.
This procedure helps mitigate the distribution shift between the true reference distribution which is unavailable, and ${\pi }_{ref}$ used by DPO.