High-Dimensional Continuous Control Using Generalized Advantage Estimation

Which action gets the credit

A trajectory ends with a number — the return — but the policy gradient needs to know which of the dozens of actions along the way deserve the praise or the blame. The object that answers this is the advantage,

$$A(s, a) = Q(s, a) - V(s),$$

which measures how much better action $a$ is than the policy's average behavior at state $s$, and so points the gradient toward actions that beat the baseline rather than merely toward actions that preceded good outcomes. The trouble is that $A$ is defined in terms of $Q$ and $V$, neither of which we know exactly, so every practical method estimates the advantage and inherits a tradeoff in how it does so.

A spectrum of estimators

The cleanest way to see the tradeoff is to bootstrap off a learned value function $V$ after a fixed number of real reward steps. The one-step estimator takes a single observed reward and then trusts $V$ for everything after:

$$\hat{A}_t^{(1)} = r_t + \gamma V(s_{t+1}) - V(s_t).$$

It has low variance because only one random action feeds into it, but high bias, because it leans almost entirely on $V$ being correct. Pushing the bootstrap further out gives the $k$-step estimator, which uses $k$ real rewards before falling back on $V$:

$$\hat{A}_t^{(k)} = \sum_{l=0}^{k-1} \gamma^l r_{t+l} + \gamma^k V(s_{t+k}) - V(s_t).$$

As $k$ grows, more of the estimate comes from observed reward and less from the value function, so bias falls while the accumulated randomness of $k$ actions drives variance up. Take $k$ to infinity and the bootstrap term vanishes entirely, leaving the Monte Carlo estimator:

$$\hat{A}_t^{(\infty)} = \sum_{l=0}^{\infty} \gamma^l r_{t+l} - V(s_t).$$

This is unbiased — it uses the actual return, not a guess — but it pays for that with the highest variance of the family, since it sums every random reward to the end of the episode.

The TD residual is the building block

Define the one-step temporal-difference residual, the gap between the bootstrapped and predicted value at a single step:

$$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t).$$

The residuals telescope: substitute the definition of each $\delta$ into a partial sum and the intermediate $V$ terms cancel, so the $k$-step advantage is exactly a discounted sum of consecutive residuals,

$$\hat{A}_t^{(k)} = \sum_{l=0}^{k-1} \gamma^l \delta_{t+l}.$$

This rewrites the whole spectrum in one currency — every $k$-step estimator is a truncated, discounted stream of $\delta$'s — which is what makes it possible to average over all of them cheaply.

Averaging over the whole spectrum

Rather than commit to a single horizon $k$, generalized advantage estimation takes an exponentially weighted average of every $k$-step estimator, with a decay $\lambda$ deciding how much weight long horizons receive. Written in residuals it collapses to a single geometric sum:

$$\hat{A}_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l \,\delta_{t+l} = (1-\lambda)\sum_{n=1}^{\infty} \lambda^{n-1}\, \hat{A}_t^{(n)}.$$

The second form makes the interpretation explicit: GAE blends the low-variance short horizons and the low-bias long horizons in fixed proportions, with $\lambda$ as the single dial that sets the mix.

The two ends of the dial

The endpoints recover the estimators we already have. At $\lambda = 0$ every term but the first drops out and GAE reduces to the one-step residual,

$$\hat{A}_t = \delta_t,$$

which is pure TD(0) — minimum variance, maximum reliance on $V$. At $\lambda = 1$ the weights stop decaying and the residuals telescope all the way to the episode's end,

$$\hat{A}_t = \sum_{l=0}^{\infty} \gamma^l r_{t+l} - V(s_t),$$

which is the Monte Carlo estimate — unbiased, highest variance. Values in between trace a continuous path from one regime to the other.

Computing it in one pass

The geometric structure means GAE never has to materialize the infinite sum. The estimate at each step is the current residual plus a discounted copy of the estimate at the next step:

$$\hat{A}_t = \delta_t + \gamma\lambda\, \hat{A}_{t+1}.$$

Sweeping backward through a trajectory from the last step to the first computes every $\hat{A}_t$ in a single linear pass, which is why GAE costs essentially nothing on top of the value-function evaluations you were already doing.

Why PPO settles near λ ≈ 0.95

PPO runs with a $V$ that is learned online and therefore imperfect, which rules out both extremes. Pure TD ($\lambda = 0$) inherits the full bias of that imperfect value function, since it trusts $V$ after a single step; full Monte Carlo ($\lambda = 1$) ignores $V$ entirely and drowns the gradient in the variance of long episodes. A setting around $\lambda = 0.95$ leans mostly on observed returns — keeping bias low where $V$ is least trustworthy — while letting the small amount of bootstrapping smooth away the worst of the variance, and empirically that is the band where continuous-control and RLHF runs train most stably.