PPO against GRPO and DPO on verifiable rewards

What I set out to check

PPO, GRPO and DPO differ mainly in where their baseline comes from. PPO learns a value function. GRPO takes the mean of eight rollouts per prompt. DPO uses the log-ratio against a reference policy. Only PPO's baseline can be *wrong* — it is an approximation, and its error feeds straight into the advantage estimate.

So the question for my half was not whether PPO beats GRPO, but where the crossover is. A small critic is cheap; is it accurate enough to be worth having, or does GRPO's baseline win by being unbiased even though it costs eight rollouts per prompt to compute?

This was a three-person study. The others built GRPO and DPO and led the head-to-head and label-regime experiments; I built PPO, the critic architectures and the advantage-error measurement, and led the critic sweep. Everything described below is that portion.

What I built

The PPO trainer and the critic study around it. Four critic capacities, behind a `build_critic()` factory so the sweep is a config change rather than a code change:

- **none** — no critic at all, REINFORCE with a batch-mean baseline - **small** — a two-layer MLP over the last-token hidden state - **medium** — a single linear head, the same depth as the policy's own LM head - **large** — a deep MLP at roughly twice the width, and twice the parameters, of medium

Alongside that, the machinery that makes the comparison measurable rather than anecdotal: advantage computation, a Monte Carlo estimate to compare against, and error metrics that turn "the critic is bad" into a quantity you can plot against accuracy. Without that, a critic sweep is just four training runs.

What went well was mostly the unglamorous part. Batched generation with left-padding, automatic bfloat16 on GPU, gradient checkpointing, and checkpoint save, load and resume with graceful handling of SIGTERM and SIGINT — which matters more than it sounds when jobs run under SLURM and can be preempted mid-run.

The per-token loss maths came out as a shared module of pure functions, so PPO and GRPO use identical clipped-surrogate and KL code without either trainer importing the other. That kept the comparison honest: neither method can benefit from a quietly different loss.

The surrounding process held up better than the experiment did. Twenty-three test files cover the trainer, the advantage maths, checkpointing, batching and the distributed paths, and the design decisions and defects live in a written spec folder — including a dated code review of my own module. Most of what follows was found by that review rather than in spite of it.

Where it diverged

The most useful thing I can report is that the experiment did not finish, and what stopped it.

On disk there are results for two of the four critic capacities — none and small — at a single seed, against a design calling for four capacities across three seeds. The head-to-head experiment produced no results at all. The one full trace that exists is a local smoke test of four rollouts. So the crossover this study was built to locate has not been measured, and the numbers that do exist say nothing about it.

The implementation has known defects too, found by review of my own module. The one that most threatens validity: prompts are built as plain text rather than through the model's chat template, which is a genuine mismatch between training and evaluation conditions for an instruction-tuned model. The reward function falls back to reading the last number in a completion, which can match an intermediate calculation instead of the final answer — a small reward-hacking surface on GSM8K. Evaluating on twenty problems during training made convergence curves too noisy to read. Policy and critic run as separate, only partly batched forward passes, which costs throughput on exactly the long runs that were needed. Multi-GPU via Accelerate or FSDP was specified and never landed, capping scale at a single device.

Those last two are most of the reason the sweep is unfinished rather than merely unfinished-looking: it was too slow to run at the scale the design asked for.

There is also a subtler problem with the measurement itself, which I would want resolved before trusting any crossover the sweep produced. Advantages are z-scored before the update, which is standard practice and reduces gradient variance — but centring subtracts the batch mean, and that removes any *constant* component of the critic's error along with it. The study's error metrics do not: mean absolute error and RMSE against a Monte Carlo baseline both include bias, and bias is recorded per capacity. So the quantity plotted on the x-axis contains something the training signal has already discarded. A critic that is badly but uniformly wrong looks bad by the metric and costs the gradient almost nothing. Whatever the sweep shows about where the critic stops paying for itself, it is measuring the shape of the error rather than its offset, and the write-up has to say which of those it means.

What I would do next

This one stopped, so the honest answer is what it would have taken rather than what is planned.

The chat template first, because every number downstream inherits that train-and-eval mismatch and nothing measured before it is safe to quote. Then the throughput work that makes the full sweep affordable at all — batching the policy and critic passes together, and the distributed training that was specified and never landed. Without those the design does not fit the compute available, which is most of why it did not finish.

The measurement question would have to be settled before the plot meant anything: either report the critic's error with its constant offset removed, matching what the normalised advantage actually responds to, or keep the offset and stop calling the result a statement about training. Choosing is more interesting than it sounds, because it decides whether the study is about the critic's accuracy or the shape of its mistakes.

What I take from it is mostly about scoping. The infrastructure was sound and the question was a good one; the design assumed compute I did not have, and no amount of engineering around the edges fixes an experiment that is too big for its cluster.