A General Theoretical Paradigm to Understand Learning from

The prevalent deployment of learning from human preferences through reinforcement learning (RLHF) relies on two important approximations: the first assumes that pairwise preferences can be substituted with pointwise rewards. The second assumes that a reward model trained on these pointwise rewards can generalize from collected data to out-of-distribution data sampled by the policy. Recently, Direct Preference Optimisation (DPO) has been proposed as an approach that bypasses the second approximation and learn directly a policy from collected data without the reward modelling stage. However, this method still heavily relies on the first approximation.

In this paper we try to gain a deeper theoretical understanding of these practical algorithms. In particular we derive a new general objective called ΨPO for learning from human preferences that is expressed in terms of pairwise preferences and therefore bypasses both approximations. This new general objective allows us to perform an in-depth analysis of the behavior of RLHF and DPO (as special cases of ΨPO) and to identify their potential pitfalls. We then consider another special case for ΨPO by setting Ψ simply to Identity, for which we can derive an efficient optimisation procedure, prove performance guarantees and demonstrate its empirical superiority to DPO on some illustrative examples.

Learning from human preferences (Christiano et al., 2017) is a paradigm adopted in the natural language processing literature to better align pretrained (Radford et al., 2018; Ramachandran et al., 2016) and instruction-tuned (Wei et al., 2022) generative language models to human desiderata. It consists in first collecting large amounts of data where each datum is composed of a context, pairs of continuations of the context, also called generations, and a pairwise human preference that indicates which generation is the best. Then, a policy generating good generations given a context is learnt from the collected data. We frame the problem of learning from human preferences as an offline contextual bandit problem (Lu et al., 2010). The goal of this bandit problem is that given a context, to choose an action (playing the role of the generation) which is most preferred by a human rater under the constraint that the resulting bandit policy should be close to some known reference policy. The constraint of staying close to a known reference policy can be satisfied e.g., by using KL regularisation (Geist et al., 2019) and its role is to avoid model drift (Lazaridou et al., 2020; Lu et al., 2020).

A prominent approach to tackle the problem of learning from human preferences is through reinforcement learning from human feedback (RLHF, Ouyang et al., 2022; Stiennon et al., 2020) in which first a reward model is trained in the form of a classifier of preferred and dispreferred actions. Then the bandit policy is trained through RL to maximize this learned reward model while minimizing the distance with the reference policy. Recently RLHF has been used successfully in solving the problem of aligning generative language models with human preferences (Ouyang et al., 2022). Furthermore recent works such as direct preference optimisation (DPO, Rafailov et al., 2023) and (SLiC-HF, Zhao et al., 2023) have shown that it is possible to optimize the bandit policy directly from human preferences without learning a reward model. They also have shown that on a selection of standard language tasks they are competitive with the state of the art RLHF while they are simpler to implement and require less resources.

Despite this practical success, little is known regarding theoretical foundations of these practical methods. Notable exceptions, that consider specific special cases, are (Wang et al., 2023; Chen et al., 2022) and prior work on preference-based (Busa-Fekete et al., 2014, 2013) and dueling bandits and RL (Novoseller et al., 2020; Pacchiano et al., 2023). However, these theoretical works focus on providing theoretical guarantees in terms of regret bounds in the standard bandit setting and they do not deal with the practical setting of RLHF, DPO and SLiC-HF. In this work, our focus is on bridging the gap between theory and practice by introducing a simple and general theoretical representation of the practical algorithms for learning from human preferences. In particular, we show that it is possible to characterise the objective functions of RLHF and DPO as special cases of a more general objective exclusively expressed in terms of pairwise preferences. We call this objective Ψ-preference optimisation (ΨPO) objective, where Ψ is an arbitrary non-decreasing mapping.

We then analyze this objective function in the special cases of RLHF and DPO and investigate its potential pitfalls. Our theoretical investigation of RLHF and DPO reveals that in principle they can be both vulnerable to overfitting. This is due to the fact that those methods rely on the strong assumption that pairwise preferences can be substituted with Elo-scores (pointwise rewards) via a Bradley-Terry (BT) modelisation (Bradley and Terry, 1952). In particular, this assumption could be problematic when the (sampled) preferences are deterministic or nearly deterministic as it leads to over-fitting to the preference dataset at the expense of ignoring the KL-regularisation term (see Sec. 4.2). We then present a simple solution to avoid the problem of overfitting, namely by setting Ψ to identity in the ΨPO. This method is called Identity-PO (IPO) and by construction bypasses the BT modelisation assumption for preferences (see Sec. 5). Finally, we propose a practical solution, via a sampled loss function (see Sec. 5.2), to optimize this simplified version of ΨPO empirically and, we compare its performance with DPO on simple bandit examples, providing empirical support for our theoretical findings (see Sec. 5.3 and Sec. 5.4).

In the remaining, we build on the notations of DPO (Rafailov et al., 2023). Given a context $x \in \mathcal{X}$, where $\mathcal{X}$ is the finite space of contexts, we assume a finite action space $\mathcal{Y}$. A policy $\pi \in \Delta_\mathcal{Y}^\mathcal{X}$ associates to each context $x \in \mathcal{X}$ a discrete probability distribution $\pi(\cdot|x) \in \Delta_\mathcal{Y}$ where $\Delta_\mathcal{Y}$ is the set of discrete distributions over $\mathcal{Y}$. We denote $\mu \in \Delta_\mathcal{Y}^\mathcal{X}$ the behavior policy. From a given context $x$, let $y, y' \sim \mu(x)$ be two actions generated independently by the reference policy. These are then presented to human raters who express preferences for one of the generations, denoted as $y_w \succ y_l$ where $y_w$ and $y_l$ denote the preferred and dispreferred actions amongst $\{y, y'\}$ respectively.

We then write true human preference $p^*(y \succ y' | x)$ the probability of $y$ being preferred to $y'$ knowing the context $x$. The probability comes from the randomness of the choice of the human we ask for their preference. So $p^*(y \succ y' | x) = \mathbb{E}_h[\mathbb{I}\{h \text{ prefers } y \text{ to } y' \text{ given } x\}]$, where the expectation is over humans $h$. We also introduce the expected preference of a generation $y$ over a distribution $\mu$ knowing $x$, noted $p^*(y \succ \mu | x)$, via the following equation:

For any two policies $\pi, \mu \in \Delta_\mathcal{Y}^\mathcal{X}$ and a context distribution $\rho$ we denote the total preference of policy $\pi$ over $\mu$ as

In practice, we do not observe $p^*$ directly, but samples $I(y, y'|x)$ from a Bernoulli distribution with mean $p^*(y \succ y' | x)$ (i.e., $I(y, y'|x)$ is $1$ with probability $p^*(y \succ y' | x)$ and $0$ otherwise). In particular, we assume we have access to the preferences through a dataset of rated generations $\mathcal{D} = (x_i, y_i, y'_i)_{i=1}^N = (x_i, y_{w,i} \succ y_{l,i})_{i=1}^N$, where $N$ is the dataset size. In addition, for a general finite set $S$, a discrete probability distribution $\eta \in \Delta_S$ and a real function $f \in \mathbb{R}^S$, we note the expectation of $f$ under $\eta$ as $\mathbb{E}_{s \sim \eta}[f(s)] = \sum_{s \in S} f(s)\eta(s)$. For a finite dataset $\mathcal{D} = (s_i)_{i=1}^N$ with $s_i \in S$, we denote the empirical expectation as $\mathbb{E}_{s \sim \mathcal{D}}[f(s)] = \frac{1}{N}\sum_{i=1}^N f(s_i)$.

3.1 Reinforcement Learning from Human Feedback (RLHF). The standard RLHF paradigm (Christiano et al., 2017; Stiennon et al., 2020) consists of two main stages: (i) learning the reward model; (ii) policy optimisation using the learned reward. 3.1.1 Learning the Reward Model. Learning a reward model consists in training a binary classifier to discriminate between the preferred and dispreferred actions using a logistic regression loss. For the classifier, a popular choice is the Bradley-Terry model: for a given context $x$ and action $y$, we denote the pointwise reward, which can also be interpreted as an Elo score, of $y$ given $x$ by $r(x, y)$. The Bradley-Terry model represents the preference function $p(y \succ y'|x)$ (classifier) as a sigmoid of the difference of rewards:

where $\sigma(\cdot)$ denotes the sigmoid function and plays the role of normalisation. Given the dataset $\mathcal{D} = (x_i, y_{w,i} \succ y_{l,i})_{i=1}^N$ one can learn the reward function by optimizing the following logistic regression loss

Assuming that $p^*(y \succ y' | x)$ conforms to the Bradley-Terry model, one can show that as the size of $\mathcal{D}$ grows, $p(y \succ y'|x)$ becomes a more and more accurate estimate of the true $p^*(y \succ y'|x)$ and in the limit converges to it.

3.1.2 Policy Optimisation with the Learned Reward. Using the reward (Elo-score) $r(x, y)$ the RLHF objective is simply to optimize for the policy $\pi \in \Delta_\mathcal{Y}^\mathcal{X}$ that maximizes the expected reward while minimizing the distance between $\pi$ and some reference policy $\pi_\mathrm{ref} \in \Delta_\mathcal{Y}^\mathcal{X}$ through the following KL-regularised objective:

in which the context $x$ is drawn from $\rho$ and the action $y$ is drawn from $\pi(\cdot|x)$. The divergence $D_\mathrm{KL}(\pi\,\|\,\pi_\mathrm{ref})$ is defined as $D_\mathrm{KL}(\pi\,\|\,\pi_\mathrm{ref}) = \mathbb{E}_{x\sim\rho}[\mathrm{KL}(\pi(\cdot|x)\,\|\,\pi_\mathrm{ref}(\cdot|x))]$ where $\mathrm{KL}(\pi(\cdot|x)\,\|\,\pi_\mathrm{ref}(\cdot|x)) = \mathbb{E}_{y\sim\pi(\cdot|x)}[\log(\pi(y|x)/\pi_\mathrm{ref}(y|x))]$. The objective in Equation (3) is essentially optimized by PPO (Schulman et al., 2017) or similar approaches. The combination of RLHF + PPO has been used with great success in practice (e.g., InstructGPT and GPT-4; Ouyang et al., 2022; OpenAI, 2023).

3.2 Direct Preference Optimisation. An alternative approach to the RL paradigm described above is direct preference optimisation (DPO; Rafailov et al., 2023), which avoids the training of a reward model altogether. The loss that DPO optimises, given an empirical dataset $\mathcal{D}$, as a function of $\pi$, is given by

In its population form, the loss takes on the form

Rafailov et al. (2023) show that when (i) the Bradley-Terry model in Equation (1) perfectly fits the preference data and (ii) the optimal reward function $r$ is obtained from the loss in Equation (2), then the global optimisers of the RLHF objective in Equation (3) and the DPO objective in Equation (5) perfectly coincide. In fact, this correspondence is true more generally; see Proposition 4 in Appendix B.

A central conceptual contribution of the paper is to propose a general objective for RLHF, based on maximising a non-linear function of preferences. To this end, we consider a general non-decreasing function $\Psi : [0, 1] \to \mathbb{R}$, a reference policy $\pi_\mathrm{ref} \in \Delta_\mathcal{Y}^\mathcal{X}$, and a real positive regularisation parameter $\tau \in \mathbb{R}^*_+$, and define the Ψ-preference optimisation objective (ΨPO) as

This objective balances the maximisation of a potentially non-linear function of preference probabilities with the KL regularisation term which encourages policies to be close to the reference $\pi_\mathrm{ref}$. This is motivated by the form of Equation (3), and we will see in the next subsection that it strictly generalises both RLHF and DPO, when the BT model holds.

4.1 A Deeper Analysis of DPO and RLHF. In the remaining, we omit the dependency on $x$ for ease of notation. This is without losing generality and all the following results are true for all $x \in \mathrm{Supp}(\rho)$. We first connect DPO and RLHF with the Ψ-preference objective in Equation (6), under the special choice of $\Psi(q) = \log(q/(1-q))$. More precisely, the following proposition establishes this connection.

Proposition 1. Suppose $\Psi(q) = \log(q/(1-q))$. When the Bradley-Terry model holds for $p^*$, that is, there exists $r : \mathcal{Y} \to \mathbb{R}$ such that $p^*(y \succ y') = \sigma(r(y) - r(y'))$, then the optimal policies for Equation (6), for the RLHF objective in Equation (3), and for the standard DPO objective in Equation (5) are identical.

Proof. Under the Bradley-Terry assumption, $\mathbb{E}_{y'\sim\mu}[\Psi(p^*(y \succ y'))] = \mathbb{E}_{y'\sim\mu}[\log(e^{r(y)}/e^{r(y')})] = r(y) - \mathbb{E}_{y'\sim\mu}[r(y')]$. This is equal to the reward in Equation (3) up to an additive constant, so the optimal policies for Equation (6) and Equation (3) coincide. By Rafailov et al. (2023) the optimal policies for Equation (5) and Equation (3) also coincide, which proves the claim.

Applying this proposition to the objective function of Equation (6), for which there exists an analytical solution, reveals that under the BT assumption the closed-form solution to DPO and RLHF can be written as

The derivation leading to Equation (7) is a well known result and is provided in App. A.1 for completeness.

4.2 Weak Regularisation and Overfitting. It is worth taking a step back, and asking what kinds of policies the above objective leads us to discover. This highly non-linear transformation of the preference probabilities means that small increases in preference probabilities already close to 1 are just as incentivised as larger increases in preference probabilities around 50%, which may be undesirable. The maximisation of logit-preferences, or Elo score in game-theoretic terminology, can also have counter-intuitive effects, even in transitive settings (Bertrand et al., 2023).

Consider the simple example where we have two actions $y$ and $y'$ such that $p^*(y \succ y') = 1$, i.e., $y$ is always preferred to $y'$. Then the Bradley-Terry model would require that $(r(y) - r(y')) \to +\infty$ to satisfy (1). If we plug this into the optimal policy (7) then we would get that $\pi^*(y')/\pi^*(y) = 0$ (i.e., $\pi^*(y') = 0$) irrespective of what constant $\tau$ is used for the KL-regularisation. Thus the strength of the KL-regularisation becomes weaker and weaker the more deterministic the preferences.

The weakness of the KL-regularisation becomes even more pronounced in the finite data regime, where we only have access to a sample estimate of the preference $\hat{p}(y \succ y')$. Even if the true preference is, e.g., $p^*(y \succ y') = 0.8$, empirically it can be very possible when we only have a few data points to estimate $\hat{p}(y \succ y') = 1$, in which case the empirical optimal policy would make $\pi(y') = 0$ for any $\tau$. This means that overfitting can be a substantial empirical issue, especially when the context and action spaces are extremely large as it is for large language models.

Why may standard RLHF be more robust to this problem in practice? While a purported advantage of DPO is that it avoids the need to fit a reward function, we observe that in practice when empirical preference probabilities are in the set $\{0, 1\}$, the reward function ends up being underfit. The optimal rewards in the presence of $\{0, 1\}$ preference probabilities are infinite, but these values are avoided, and indeed regularisation of the reward function has been observed to be an important aspect of RLHF training in practice (Christiano et al., 2017). This underfitting of the reward function is thus crucial in obtaining a final policy that is sufficiently regularised towards the reference policy $\pi_\mathrm{ref}$, and DPO, in avoiding the training of the reward function, loses the regularisation of the policy that the underfitted reward function affords.

While standard empirical practices such as early-stopping can still be used as an additional form of regularisation to curtail this kind of overfitting, in the next section, we will introduce a modification of the ΨPO objective such that the optimal empirical policy can be close to $\pi_\mathrm{ref}$ even when preferences are deterministic.

We have observed in the previous section that DPO is prone to overfitting, and this stems from a combination of the unboundedness of $\Psi$, together with not training an explicit reward function. Not training a reward function directly is a clear advantage of DPO, but we would like to avoid the problems of overfitting as well. This analysis of DPO motivates choices of $\Psi$ which are bounded, ensuring that the KL regularisation in Equation (6) remains effective even in the regime of $\{0, 1\}$-valued preferences, as it is often the case when working with empirical datasets. A particularly natural form of objective to consider is given by taking $\Psi$ to be the identity mapping in Equation (6), leading to direct regularised optimisation of total preferences:

The standard approach to optimise the objective function of Equation (8) is through RLHF with the choice of reward $r(y) = p^*(y \succ \mu)$. However both using RL and estimating the reward model $r(y)$ can be costly. Inspired by DPO one would like to devise an empirical solution for the optimisation problem of Equation (8) which can directly learn from the preference dataset. Thus it would be able to avoid RL and reward modelling altogether.

5.1 Derivations and Computationally Efficient Algorithm. As with DPO, it will be beneficial to re-express Equation (8) as an offline learning objective. To derive such an expression, we begin by following the derivation of Rafailov et al. (2023), manipulating the analytic expression for the optimal policy into a system of root-finding problems. As in the previous section, we drop dependence on the context $x$ from our notation, as all arguments can be applied on a per-context basis.

Root-finding problems. Let $g(y) = \mathbb{E}_{y'\sim\mu}[\Psi(p^*(y \succ y'))]$. Then we have

For any $y, y' \in \mathrm{Supp}(\pi_\mathrm{ref})$, we therefore have

By letting $h^*(y, y') = \log\!\Bigl(\frac{\pi^*(y)\pi_\mathrm{ref}(y')}{\pi^*(y')\pi_\mathrm{ref}(y)}\Bigr)$ and rearranging Equation (10), we obtain

The core idea now is to consider a policy $\pi$, define $h_\pi(y, y') = \log\!\Bigl(\frac{\pi(y)\pi_\mathrm{ref}(y')}{\pi(y')\pi_\mathrm{ref}(y)}\Bigr)$, and aim to solve the equations $h_\pi(y, y') = \tau^{-1}(g(y) - g(y'))$, namely Equation (12).

Loss for IPO. We now depart from the approach to the analysis employed by Rafailov et al. (2023), to obtain a novel offline formulation of Equation (6), in the specific case of $\Psi$ as the identity function. In this case, Equation (12) reduces to

We begin by re-expressing these root-finding problems as a single optimisation problem $L(\pi)$:

One can easily show that for the choice of $\pi^*$ we have $L(\pi^*) = 0$. Thus $\pi^*$ is a global minimiser of $L(\pi)$. The following theorem establishes the uniqueness of this solution.

Theorem 2 (Uniqueness of Global/Local Optima). Assume that $\mathrm{Supp}(\mu) = \mathrm{Supp}(\pi_\mathrm{ref})$ and define $\Pi$ to be the set of policies $\pi$ such that $\mathrm{Supp}(\pi) = \mathrm{Supp}(\mu)$. Then $\pi \mapsto L(\pi)$ has a unique local/global minimum in $\Pi$, which is $\pi^*$.

Proof sketch. By assumption $\pi^* \in \Pi$, and by definition $\forall \pi \in \Pi$, $L(\pi) \geq 0$. From Equation (11), $L(\pi^*) = 0$, so $\pi^*$ is a global optimum. We parametrise $\Pi$ via vectors of logits $s \in \mathbb{R}^\mathcal{J}$ with $\mathcal{J} = \mathrm{Supp}(\mu)$, setting $\pi_s(y) = e^{s(y)} / \sum_{y' \in \mathcal{J}} e^{s(y')}$. Writing $L(s) = L(\pi_s)$,

The objective is quadratic in $s$. Expanding it, the loss decomposes into a sum of squares

plus linear and constant terms, which is a positive-semidefinite quadratic, hence convex (Boyd & Vandenberghe, 2004, Chap. 4). The only direction in which (15) does not increase away from $0$ is the $(1,\ldots,1)$ direction, in which $\pi_s$ does not change. Hence $L$ is strictly convex on the quotient space and $\pi^*$ is the unique local/global minimum in $\Pi$.

5.2 Sampled Loss for IPO. In order to obtain the sampled loss for IPO we need to show that we can build an unbiased estimate of the right-hand side of Equation (13). To this end, we consider the Population IPO Loss:

where $I(y, y')$ is drawn from a Bernoulli distribution with mean $p^*(y \succ y')$, i.e., $I(y, y')$ is $1$ if $y$ is preferred to $y'$ (which happens with probability $p^*(y \succ y')$), and $0$ otherwise. This straightforwardly yields a sample-based loss that can be used, by sampling a pair $(y, y')$ from the preference dataset and consulting the recorded preference to obtain a sample from $I(y, y')$.

Proposition 3. The expressions in Equation (13) and Equation (16) are equal, up to an additive constant independent of $\pi$.

We now discuss how to approximate the loss in Equation (16) with an empirical dataset. As in our earlier discussion, the empirical dataset $\mathcal{D}$ takes the form $(y_{w,i}, y_{l,i})_{i=1}^N$. Note that each datapoint $(y_{w,i}, y_{l,i})$ contributes two terms to an empirical approximation of Equation (16), with $(y, y', I(y, y')) = (y_{w,i}, y_{l,i}, 1)$, and also $(y, y', I(y, y')) = (y_{l,i}, y_{w,i}, 0)$. This symmetry is important to exploit, and leads to a reduction in the variance of the loss. The overall empirical loss is therefore given by

which up to a constant equals

This simplified form of the loss provides some valuable insights on the way in which IPO optimises the policy $\pi$: IPO learns from the preference dataset simply by regressing the gap between log-likelihood ratios $\log(\pi(y_w)/\pi(y_l))$ and $\log(\pi_\mathrm{ref}(y_w)/\pi_\mathrm{ref}(y_l))$ to $\tau^{-1}/2$. So the weaker the regularisation becomes, the higher would be the log-likelihood ratio of $y_w$ to $y_l$. In other words IPO, unlike DPO, always regularises its solution towards $\pi_\mathrm{ref}$ by controlling the gap between the log-likelihood ratios, thus avoiding the over-fitting to the preference dataset.

We summarise the sampled IPO in Algorithm 1: given a dataset $\mathcal{D}$ of prompts and preferred/dispreferred generations $(x, y_w, y_l)$ and a reference policy $\pi_\mathrm{ref}$, define $h_\pi(y, y', x) = \log\!\bigl(\frac{\pi(y|x)\pi_\mathrm{ref}(y'|x)}{\pi(y'|x)\pi_\mathrm{ref}(y|x)}\bigr)$ and, starting from $\pi = \pi_\mathrm{ref}$, minimise

5.3 Illustrative Examples — Asymptotic Setting. To illustrate the qualitative difference between our algorithm and DPO we consider a few simple cases. For simplicity we assume there is no context $x$, i.e., we are in the bandit setting. We first consider the simple case where we have $2$ actions only, $y_1$ and $y_2$, and a deterministic preference between them: $p^*(y_1 \succ y_2) = 1$. Suppose we start with a uniform $\pi_\mathrm{ref}$ and $\mu$. We know from Section 4.2 that DPO will converge to the deterministic policy $\pi^*(y_1) = 1$, $\pi^*(y_2) = 0$ regardless of the value of $\tau$. Thus even when the regularisation coefficient $\tau$ is very large, this is very different from the uniform $\pi_\mathrm{ref}$.

Now, let us derive the optimal policy for IPO. We have $p^*(y_1 \succ \mu) = 3/4$ and $p^*(y_2 \succ \mu) = 1/4$. Plugging this into Equation (9) with $\Psi = I$ gives

Hence we see that if we have large regularisation as $\tau \to +\infty$, then $\pi^*$ converges to the uniform policy $\pi_\mathrm{ref}$, and on the flip side as $\tau \to 0$, then $\pi^*(y_1) \to 1$ and $\pi^*(y_2) \to 0$, which is the deterministic optimal policy. The regularisation parameter $\tau$ can now actually be used to control how close to $\pi_\mathrm{ref}$ we are.

5.4 Sampled Preferences. So far we relied on the closed-form optimal policy from Eq. (9) to study DPO and IPO's stability, but this equation is not applicable to more complex settings where we only have access to sampled preferences instead of $p^*$. We can still however find accurate approximations of the optimal policy by choosing a parametrisation $\pi_\theta$ and optimising $\theta$ with an empirical loss over a dataset and iterative gradient-based updates. We use this approach to show two non-asymptotic examples where DPO over-fits the dataset of preferences and ignores $\pi_\mathrm{ref}$: when one action $y$ wins against all others DPO pushes $\pi_\theta(y)$ to $1$ regardless of $\tau$, and conversely when one action $y$ never wins against the others DPO pushes $\pi_\theta(y)$ to $0$ again regardless of $\tau$. In the same scenarios, IPO does not converge to these degenerate solutions but instead remains close to $\pi_\mathrm{ref}$ based on the strength of the regularisation $\tau$.

For both scenarios we consider a discrete space $\mathcal{Y} = \{y_a, y_b, y_c\}$ with $3$ actions, and select a dataset of pairs $\mathcal{D} = \{(y_{w,i}, y_{l,i})\}$. Given $\mathcal{D}$, we leverage the empirical losses from Eq. (4) and Eq. (13) to find DPO's and IPO's optimal policy. We encode policies as $\pi_\theta(y_i) = \mathrm{softmax}(\theta)_i$ using a vector $\theta \in \mathbb{R}^3$, and optimise them for $18000$ steps using Adam (Kingma and Ba, 2014) with learning rate $0.01$ and mini-batch size $9$. Mini-batches are constructed using uniform sampling with replacement from $\mathcal{D}$. Both policies and losses are implemented using the flax python framework (Bradbury et al., 2018; Heek et al., 2023), and the Adam implementation is from optax (Babuschkin et al., 2020). For each set of hyperparameters we repeat the experiment $10$ times with different seeds, and report mean and $95\%$ confidence intervals.

IPO Avoids Greedy Policies. For the first example we sample each unique action pair once to collect a dataset $\mathcal{D}$ containing $3$ observed preferences. Due to symmetries of pairwise preferences, sampling only $3$ preferences can result in only two outcomes (up to permutations of the actions): $\mathcal{D}_1 = \{(y_a, y_b), (y_b, y_c), (y_a, y_c)\}$ and $\mathcal{D}_2 = \{(y_a, y_b), (y_b, y_c), (y_c, y_a)\}$, where we focus on $\mathcal{D}_1$, which represents a total ordering, rather than $\mathcal{D}_2$, which represents a cycle. The outcome of the experiment is reported in Fig. 1 in which we plot the learning curves for varying values of $\tau$. We observe that DPO always converges to the deterministic policy for all values of $\tau$. In other words DPO completely ignores the reference policy, no matter how strong is the regularisation term, and converges to the action which is preferred in the dataset. On the other hand, IPO prevents the policy from becoming greedy when the regularisation is strong.

IPO Does not Exclude Actions. In the first example DPO converges to a deterministic policy because one action strictly dominates all others and the loss continues to push up its likelihood until it saturates. The opposite effect happens for the logical opposite condition, i.e., when one action does not have at least a victory in the dataset DPO will set its probability to $0$ regardless of $\tau$. While this is less disruptive than the first example (a single probability is perturbed whereas previously the whole policy was warped by an over-achieving action) it is also much more common in real-world data. In particular, whenever the action space is large but the dataset small, some actions will necessarily be sampled rarely or only once, making it likely to never observe a victory. Especially because we do not have data on their performance, $\pi$ should stick close to $\pi_\mathrm{ref}$ for safety, but DPO's objective does not promote this.

In the final example the dataset consists of two observed preferences $\mathcal{D}_3 = \{(y_a, y_b), (y_b, y_a)\}$ and leaves the pair $(y_a, y_c)$ completely unobserved. We compute solutions using Adam once again, and report the results in Fig. 2 for varying values of $\tau$. We observe again here that DPO ignores the prior $\pi_\mathrm{ref}$ completely, no matter how strong we regularise the objective, whereas IPO gradually decreases the probability of the unobserved action with $\tau$.

6 Conclusion and Future Work. We presented a unified objective, called ΨPO, for learning from preferences. It unifies RLHF and DPO methods. In addition, we introduced a particular case of ΨPO, called IPO, that allows to learn directly from preferences without a reward modelling stage and without relying on the Bradley-Terry modelisation assumption that assumes that pairwise preferences can be substituted with pointwise rewards. This is important because it allows to avoid the overfitting problem. This theoretical contribution is only useful in practice if an empirical sampled loss function can be derived. This is what we have done in Sec. 5 where we show that IPO can be formulated as a root-finding problem from which an empirical sampled loss function can be derived. The IPO loss function is simple, easy to implement and theoretically justified. Finally, in Sec. 5.3 and Sec. 5.4, we provide illustrative examples where we highlight the instabilities of DPO when the preferences are fully-known as well as when they are sampled. Those minimal experiments are sufficient to prove that IPO is better suited to learn from sampled preferences than DPO. Future works should scale those experiments to more complex settings such as training language models on human preferences data.