Reinforcement Learning For Multi-Turn Agents

Why single-turn methods break

The credit-assignment problem. Reward arrives only after the whole interaction. Which of the twelve actions actually mattered? Trajectory-level reward treats the entire path as one decision and can't tell. The partition-function mismatch. DPO's math relies on a normalization constant cancelling out — true only when it depends on a static prompt. In a multi-turn setting it depends on the ever-changing state, so it doesn't cancel, and standard DPO breaks with a length-dependent bias. Compounding errors. Behavioral cloning forces the agent down expert paths; one small deviation lands it in an unseen state, and failures cascade.

Part 3 of our series on model alignment. Earlier posts covered gold labels vs. preference pairs vs. verifiers and aligning model behavior the way humans like.

DPO's derivation contains one line of algebra that everything else depends on: the partition function — the normalization constant that would otherwise require summing over every possible response — cancels out of the loss. That cancellation is the entire reason DPO is cheap. It is also only valid when the constant depends on a single, static prompt. Hand the model a browser, a terminal, or a twelve-turn dialogue, and the state changes after every action. The constant stops being constant, it stops cancelling, and the loss you're optimizing is no longer the loss the paper derived.

This is worth being precise about, because the usual framing gets it wrong. Teams don't discover that DPO "doesn't scale to agents." They discover that vanilla DPO was never mathematically valid in a multi-turn setting to begin with — what's left after the broken cancellation is a length-dependent bias term, so training silently learns to prefer trajectories by length rather than by quality. The method didn't degrade. Its assumptions were violated from step one.

Two ways to create RL for Agents

1)DMPO: Make the partition function constant again

• DMPO — Direct Multi-Turn Preference Optimization, from USTC and Meta AI — redesigns DPO's core math for multi-turn environments rather than patching around it. The central move: swap DPO's policy constraint for a state-action occupancy measure (SAOM) constraint in the RL objective. Under the SAOM constraint, the troublesome partition function becomes constant across states and cancels out cleanly again. The derivation goes through; you're back to a simple classification-style loss, now valid over trajectories.

Two further pieces make it trainable in practice. First, length normalization inside the Bradley–Terry model, so wildly mismatched trajectory lengths — a 4-turn success paired against a 15-turn failure — stop breaking training. Second, the SAOM constraint itself doubles as error correction: it actively pulls a wandering agent back toward states the successful trajectories occupied, which directly attacks the compounding-error cascade that behavioral cloning suffers.

The pipeline runs in four phases:

1. SFT a base model (Llama-2-7B-Chat, Mistral-7B) on expert trajectories. This sets π_ref — the reference policy everything else is measured against.
2. Roll out that model to gather both good and flawed paths, and assemble them into preference pairs.
3. Set β ∈ [0.1, 0.9] and the discount γ ∈ [0.1, 0.99]. The γ choice is the one that carries information about your data: use low γ for noisy data that tends to fail early, high γ for clean data that fails late. Getting this backwards means discounting exactly the turns where your failures live.
4. Optimize with AdamW — cosine schedule, 3% warmup, batch size 32, learning rate swept over {1e-5, 2e-5, 3e-5}.

Nothing exotic anywhere in that list. That's the point: the innovation is in the objective, and once the objective is right, the training recipe is the boring one you already know.

• PNLC: don't fine-tune the big model at all

PNLC — "Planning without Search," built on offline goal-conditioned RL — takes a different tack entirely: leave the frontier model's weights alone.

Instead, train a tiny MLP critic offline with Implicit Q-Learning over text-embedded (state, thought, goal) tuples. All the RL machinery — the value learning, the credit assignment — lives in a network small enough to train on a workstation, over embeddings rather than tokens.

At inference, the division of labor inverts the usual picture. The frontier LLM proposes a thought, then hallucinates four hypothetical subgoals — two good, two bad — and scores all four through the MLP critic. From that data-driven feedback it self-refines its plan. The LLM supplies imagination; the small critic supplies judgment about which imagined futures are actually reachable and valuable.

Hallucination, normally the failure mode, is repurposed as the exploration mechanism — the critic is what keeps it honest.

The practical appeal is hard to overstate if you're working against a tight budget or a closed API. There's no fine-tuning bill, no serving a custom checkpoint, and the critic retrains cheaply as your environment shifts. The honest caveat: the critic is only as good as the offline trajectories it learned from, and a goal distribution that drifts away from that data will degrade it silently — the standard offline-RL risk, relocated but not removed.

Final decision Tree for model alignment

If you're building an agent and deciding how to train it, the fork is fairly clean.

Reach for DMPO when you can fine-tune, you can collect trajectory-level preference pairs, and you want the improvement baked into the weights. Budget your care for the γ decision — it's the one hyperparameter in the recipe that encodes an actual belief about where your trajectories fail, and it's cheap to get wrong and expensive to notice.

Reach for PNLC when the base model is frozen — a closed API, a compliance constraint, a compute budget that rules out 7B-scale fine-tuning — or when your environment changes faster than you're willing to re-run a training pipeline. Retraining a small MLP beats re-running SFT-plus-preference-optimization every time the task shifts.

And in either case, the foundation doesn't change: SFT on expert trajectories comes first. DMPO requires it explicitly to set π_ref; PNLC assumes a model already competent enough to propose plausible thoughts and subgoals. As the decision tree puts it — many turns → agent RL, and SFT is the floor every one of these methods stands on.

Still open question?

DMPO and PNLC repair single-turn assumptions; neither dissolves the deeper problem. Credit assignment at the level of individual turns — knowing that turn 3 saved the episode and turn 9 was waste — is still coarse in both: DMPO scores whole trajectories against each other, and PNLC's critic evaluates subgoals, not the fine-grained actions between them. The methods multiplying in this space — MT-GRPO, GTPO, Turn-PPO, GiGPO — are largely attempts to push credit down to the turn level, and none has yet become the default the way GRPO did for verifiable single-turn tasks.

The durable lesson from this round is narrower and more useful: before scaling an alignment method into a new setting, find the line in its derivation where something cancels, and check whether it still does. That single check would have predicted everything in this post.