Individual contribution · Off-policy RL

Benchmarking Off-Policy Reinforcement Learning Controllers for Quadcopter Stabilisation

Wietske van der Westhuizen · Supervisor: Krupa Prag · University of Cape Town

SACTD3TAD3PID

Final report (PDF) Literature review (PDF) Source code (ZIP)

Summary

What I did and what I found

I compared three off-policy RL controllers, Soft Actor-Critic (SAC), Twin Delayed DDPG (TD3) and Twin Average Delayed DDPG (TAD3), against the PID controller for quadcopter hover under wind. Each was tuned with the same Optuna-driven HPO pipeline and benchmarked over 10 seeds in still air and composite wind, using flight-specific metrics and real-time latency profiling.

PID was competitive in still air but its success rate fell to 0% under composite wind. SAC was the best RL controller, while the TAD3 mean-aggregation hypothesis was inconclusive: there was not enough evidence that it differs from TD3.

83%SAC success under composite wind (PID: 0%)

0.006 mSAC still-air RMSE (PID: 0.023 m)

2.9 msSAC inference latency vs a 4 ms budget

p = 0.0039SAC vs TD3/TAD3 under wind (Holm-significant)

Goals

Research questions

RQ1Solution quality. Under a matched HPO budget and identical dynamics, do SAC, TD3 and TAD3 achieve higher hover success, lower RMSE and tighter attitude stability than PID, in both nominal and composite wind?

RQ2Algorithmic efficiency. Does replacing TD3’s minimum target critic with an average (TAD3) reduce over-conservative value estimates, and how does that affect convergence reliability and disturbance rejection?

RQ3Computational feasibility. Can the off-policy neural controllers keep inference latency below 4,000 µs, the budget for 250–1000 Hz control loops?

Method

My technical contributions

Custom TAD3 implementation

TD3 uses the minimum of its twin critics, which prevents overestimation but introduces underestimation bias. TAD3 replaces the minimum with the mean, keeping delayed policy updates and target-policy smoothing. It is implemented by subclassing SBX’s TD3 and overriding the target-Q computation.

y_{TAD3} = r + γ \frac{Q_{θ_{1}^{'}} (s^{'}, \tilde{a}) + Q_{θ_{2}^{'}} (s^{'}, \tilde{a})}{2}

Multi-stage HPO and evaluation

Calibration: Spearman rank correlation across checkpoints chose the objective (fraction in tolerance, ρ = 0.957), trial budget and episode count.
Search: 100 Optuna TPE trials per controller (20 random + 80 TPE) at 750 K steps; top 20 clustered by k-medoids into 5 candidates using a conservative score (mean minus standard error).
Validation: 5 configurations × 5 fresh seeds at 1 M steps; a four-tier rule picks the winner.
Benchmark: 10 fresh seeds, 1.5 M steps, 11 checkpoints, 3 episodes per seed-condition.

Disturbance stress-testing

Still air versus the composite wind profile (constant 0.20 hover-thrust force plus 0.15–0.40 gusts; up to 88% of available thrust to counter). Evaluation seeds are strictly disjoint from tuning and training seeds.

Latency profiling

Average wall-clock time to compute one action, measured per controller and compared with the 4,000 µs onboard-feasibility budget.

Final tuned hyperparameters (Appendix A)

**Selected configurations** after the HPO pipeline. Replay buffer 1,000,000 and learning start 10,000 steps for all.
Parameter	SAC	TD3	TAD3
Learning rate	1.49×10⁻⁴	1.18×10⁻³	2.31×10⁻³
Discount γ	0.9768	0.9771	0.9771
Polyak τ	0.0181	0.0152	0.0195
Batch size	128	512	512
Network	[256, 256]	[256, 256]	[256, 256]
Action noise	normal, 0.359	ou, 0.051	ou, 0.152
Entropy α	0.01	–	–
Policy delay	–	3	3
Target policy noise / clip	–	0.266 / 0.393	0.153 / 0.337

Results

Controller flight performance

**Benchmark summary** (10 seeds × 3 episodes per condition; mean ± SE). Lower is better for RMSE, oscillation and smoothness.
Controller	Success	Completion	RMSE last 10 s (m)	Attitude osc. (°)	Ctrl smoothness
No disturbance
SAC	1.000 ± 0.000	1.000 ± 0.000	0.006 ± 0.001	0.01 ± 0.00	2.6 ± 0.3
TD3	0.967 ± 0.033	0.967 ± 0.033	0.016 ± 0.004	0.94 ± 0.34	14.9 ± 2.6
TAD3	0.800 ± 0.133	0.900 ± 0.100	0.188 ± 0.163	7.54 ± 3.60	219.8 ± 72.4
PID	1.000 ± 0.000	1.000 ± 0.000	0.023 ± 0.001	1.02 ± 0.10	687.2 ± 27.0
Composite wind
SAC	0.833 ± 0.056	1.000 ± 0.000	0.132 ± 0.011	29.0 ± 1.5	21.3 ± 1.4
TD3	0.333 ± 0.070	0.767 ± 0.087	0.283 ± 0.040	31.6 ± 1.9	578.8 ± 123.9
TAD3	0.167 ± 0.056	0.733 ± 0.109	0.434 ± 0.137	31.9 ± 4.9	1680.4 ± 321.0
PID	0.000 ± 0.000	0.133 ± 0.074	0.426 ± 0.035	57.3 ± 1.6	483.8 ± 15.0

PID’s composite RMSE is computed over only its few completed episodes, so it flatters PID; unconditional RMSE across all episodes is 0.770 m for PID vs 0.132 m for SAC.

PID versus RL

Still air. SAC and PID both succeeded in every episode. SAC had the lowest RMSE (0.006 m vs PID’s 0.023 m; Wilcoxon p = 0.002, effect size r = −1.00, Holm-significant). TD3 was numerically better than PID but not significantly (p = 0.106). PID gave by far the most erratic control inputs (smoothness 687.2 vs SAC’s 2.6), overshooting repeatedly because its fixed-gain braking is applied identically everywhere.

Composite wind. PID’s completion rate collapsed from 100% to 13.3% and none of its episodes succeeded. Its corrective thrust trailed the gusts (phase lag), producing large elliptical swings that left the arena bounds. SAC kept 100% completion. Because PID completed so few episodes, the RL-vs-PID tests have only n = 3 pairs, so the smallest attainable p is 0.25. These are underpowered, not merely non-significant, and are read directionally.

Four 3-D plots of flown trajectories from the start point to the hover target under composite wind. SAC and TD3 settle close to the target, TAD3 oscillates more severely, and PID swings in large sweeps across the arena. — **Figure 1.** Flown trajectories of (a) SAC, (b) TD3, (c) TAD3 and (d) PID from start to the fixed hover target (0, 0, 1.5) m under composite wind (seed 202, episode 0). Panels (a)–(b) and (c)–(d) use different scales to make the trajectories visible.

SAC versus TD3 versus TAD3

**Pairwise comparison of RL controllers** on RMSE (last 10 s), paired by seed. Wilcoxon signed-rank, Holm-corrected α = 0.00417.
Comparison	Condition	Mean diff. (m)	Effect size r	p	Significant
SAC vs TD3	No disturbance	−0.010	−0.85	0.0137	No
SAC vs TAD3	No disturbance	−0.183	−1.00	0.0039	Yes
TD3 vs TAD3	No disturbance	−0.172	−0.56	0.1641	No
SAC vs TD3	Composite	−0.151	−0.96	0.0039	Yes
SAC vs TAD3	Composite	−0.305	−1.00	0.0039	Yes
TD3 vs TAD3	Composite	−0.137	−0.16	0.7344	No

SAC is the superior RL controller. It had the lowest RMSE, least attitude oscillation, smoothest inputs and highest success under both conditions. A plausible explanation is architectural: SAC’s entropy-regularised stochastic policy smooths out poor Q-estimates and adapts exploration, whereas TD3/TAD3 rely on a fixed exploration-noise hyperparameter.

The TAD3 hypothesis is inconclusive. TD3 and TAD3 were not significantly different on RMSE (p = 0.164 and 0.734). Descriptively, however, TD3 had about twice TAD3’s composite success rate and about a third of its control roughness, so mean aggregation did not demonstrably help and may have hurt by letting less-verified actions through during training.

Training dynamics

Sample efficiency

Calibration

Fraction in tolerance was the best ranking metric (ρ = 0.957).
Reward plateaued at 400 K steps (TD3), 500 K (SAC) and 750 K (TAD3), which set the 750 K search budget.

Steps needed to beat PID

TD3 beat PID’s still-air RMSE within 400 K steps, and under wind first at 900 K.
SAC took longer (1.2 M still air, 1.1 M wind) but plateaued at a higher ceiling.
TAD3 never beat PID’s still-air RMSE and was the least efficient.

Two line charts of RMSE over the final 10 seconds against training steps for SAC, TD3 and TAD3, with a dashed PID baseline: (a) no disturbance and (b) composite wind. RMSE falls as training progresses, with SAC lowest at the end under composite wind. — **Figure 2.** RMSE (final 10 s) vs training step for SAC, TD3 and TAD3 in (a) no disturbance and (b) composite wind, with ±1 SE ribbons across the 10 benchmark seeds. The dashed line is the final PID value.

Two line charts of episode completion rate against training steps for SAC, TD3 and TAD3, with a dashed PID baseline: (a) no disturbance, where SAC and TD3 reach 1.0 early, and (b) composite wind, where completion rises late in training and SAC ends at 1.0. — **Figure 3.** Completion rate vs training step, with the same controllers, conditions and PID reference as Figure 2.

Feasibility

Inference latency (RQ3)

**Mean inference latency** pooled across conditions and completed episodes.
Controller	Latency (µs)	% of 4,000 µs
SAC	2,898 ± 429	72%
TD3	2,570 ± 514	64%
TAD3	2,861 ± 605	72%
PID	44.1 ± 0.3	1.1%

All three RL controllers fit inside the real-time budget on the benchmarking machine, but PID is about 60–65× faster thanks to its O(1) arithmetic, versus full neural-network forward passes for the RL policies.

The latency is dominated by a single-sample GPU dispatch, not the algorithm, and was not measured on an embedded processor. The spread is large relative to the headroom, so feasibility on real flight hardware remains unconfirmed.

Media

Flights

Video 1. PID (left) and SAC (right) flying the same composite-wind episode: benchmark seed 200, episode 0, RL policy at 1.5 M training steps, identical spawn and wind. The green sphere is the 0.3 m success zone. Simulated in MuJoCo Warp.

Discussion

Limitations and future work

Limitations

Single airframe, simulation only; no physical flight.
3 of 12 RMSE tests are underpowered under composite wind.
Significance testing covers RMSE only; other metrics are descriptive.
Episode boundaries cut off recoverable flights for TAD3 and PID.

Future work

More seeds, conditions and airframes for a firmer ranking.
Cascade controller architectures (separate altitude and attitude loops).
Wider environment bounds to allow recovery and higher completion.
Latency on embedded hardware.

Documents

Reports and code

PDF
Final report
Benchmarking Off-Policy Reinforcement Learning Controllers for Quadcopter Stabilisation
Open
PDF
Literature review
Evaluating Off-Policy Reinforcement Learning Control Systems Against PID for Quadcopters (5 pages).
Open
ZIP
Source code
Shared project codebase (both authors).
Download

All project documents →