Department of Computer Science · University of Cape Town · Honours Project 2026
Benchmarking Reinforcement Learning Controllers for Quadcopter Stabilisation
A shared-simulation benchmark of PID and six deep reinforcement learning controllers for quadcopter hover stabilisation under wind disturbance.
At a glance
The project in four numbers
Background
Why quadcopter control is hard
Quadcopters are underactuated: six degrees of freedom, only four rotors. Their rotor dynamics are tightly coupled, they are open-loop unstable, and wind, motor noise or a shifting payload keeps nudging them away from equilibrium. As drones move into logistics, search and rescue and precision agriculture, reliable low-level control becomes the foundation everything else rests on.
The dominant solution is the Proportional–Integral–Derivative (PID) controller. It is cheap, interpretable and well understood, and it works very well near hover where the dynamics are almost linear. Its weakness is that fixed gains are tuned for ideal conditions; gusts push the vehicle into non-linear behaviour that PID was never designed for.
Reinforcement learning (RL) offers a model-free alternative: a policy learns directly from interaction with the physics. The open question is whether it genuinely beats PID, or only looks better because of how each study was set up.
The benchmarking gap we address
Undisclosed tuning
Many studies don’t report how RL or PID were tuned, so results are hard to reproduce.
Different simulators
MuJoCo, Gazebo and custom engines are mixed, so gains may be simulator artefacts, not algorithmic ones.
Wrong metrics
Reward curves dominate; hover stability, settling and steady-state error are rarely measured.
Approach
One fair arena for every controller
Every controller flies the same Bitcraze Crazyflie 2 model in the same MuJoCo Warp / Gymnasium environment, with the same reward, curriculum, wind model, tuning protocol and evaluation metrics.
Simulation environment
- Physics
- MuJoCo Warp, 500 Hz
- Control rate
- 50 Hz
- Task
- Take off from the floor, climb to 1.5 m and hover for 20 s
- Observation
- 17-D: position error, velocity, attitude, angular rate, previous action, in-tolerance flag
- Action
- 4-D normalised motor commands in [−1, 1]
Bounded multiplicative reward
Proximity gates everything. Penalties can only shrink the reward, never make it negative, so crashing early can never pay off.
with proximity F(e) = tol / (tol + e) and a 0.30 m tolerance.
The wind: still air versus composite disturbance
Controllers are evaluated in still air (best case) and under a composite profile: a constant force plus random gusts, which tests both steady-state rejection and recovery. During training the wind intensity ramps up with the curriculum, reaching 1.2× the evaluation intensity. See an example episode in motion.
| Component | Specification |
|---|---|
| Constant force | 0.20 of hover thrust |
| Gust magnitude | uniform [0.15, 0.40] per gust |
| Gust duration | uniform [0.5, 2.0] s |
| Gust interval | uniform [3.0, 5.0] s between gusts; first gust at t = 5 s |
| Direction | random, drawn once per episode and shared by the constant force and all gusts |
Fair-tuning pipeline
Calibration
Probe which metric, budget and pruning rule rank configurations most reliably.
HPO search
100 Optuna TPE trials per controller, PID included, on a matched budget.
Validation
Top 5 configurations retrained on fresh seeds; one winner picked by a fixed rule.
Benchmark
10 untouched seeds, 3 episodes per seed and condition, Wilcoxon tests with Holm correction.
Metrics
Flight-specific rather than theoretical reinforcememt learning metrics: success and completion rate, position RMSE, attitude oscillation, control smoothness, time-to-arrival and fraction-in-tolerance, plus learning curves and inference latency.
Composite wind
What the quadcopter is up against
A steady push plus unpredictable gusts, redrawn for every episode. Pick an example and watch the force on the vehicle (bottom) and the drone being shoved off its target and tilting back into it (top).
| Component | Starts | Lasts | Adds | Total |
|---|---|---|---|---|
| Constant | 0 s | whole flight | 0.20 | 0.20 |
| Gust 1 | 5.0 s | 1.4 s | 0.30 | 0.50 |
| Gust 2 | 10.2 s | 0.7 s | 0.38 | 0.58 |
| Gust 3 | 14.8 s | 1.8 s | 0.18 | 0.38 |
| Component | Starts | Lasts | Adds | Total |
|---|---|---|---|---|
| Constant | 0 s | whole flight | 0.20 | 0.20 |
| Gust 1 | 5.0 s | 0.9 s | 0.36 | 0.56 |
| Gust 2 | 10.3 s | 1.9 s | 0.21 | 0.41 |
| Gust 3 | 15.5 s | 1.2 s | 0.27 | 0.47 |
| Component | Starts | Lasts | Adds | Total |
|---|---|---|---|---|
| Constant | 0 s | whole flight | 0.20 | 0.20 |
| Gust 1 | 5.0 s | 0.6 s | 0.22 | 0.42 |
| Gust 2 | 8.8 s | 1.5 s | 0.33 | 0.53 |
| Gust 3 | 13.9 s | 0.8 s | 0.16 | 0.36 |
| Gust 4 | 18.8 s | 1.1 s | 0.39 | 0.59 |
Why this is hard
- The steady 0.20 force never goes away, so a controller must lean into it permanently.
- Gusts arrive at unpredictable times and sizes, so there is no time to retune.
- The full envelope reaches 0.60 of hover thrust, so countering it can use about 88% of the thrust available.
Our contributions
Two halves of one benchmark
The controllers are split by algorithm family. Each page has the detailed methods, results and documents for that half.
Wietske van der Westhuizen
Off-policy controllers
SACTD3TAD3PID
Includes a custom TAD3 implementation (average twin-critic target) and real-time inference-latency profiling against a 4 ms budget.
View results →Mathew Maartens
On-policy controllers
A2CPPOSCR-PPOPID
Includes a matched 30-seed ablation testing whether SCR-PPO’s dynamic clipping beats a fixed clip.
View results →Results
Wind is where learned control pulls ahead
In still air, PID is competitive with the best RL controllers. Under composite wind, PID loses almost every flight in both studies, while the strongest RL controllers keep hovering.
Success rate: composite wind
Share of evaluation episodes that took off and held a stable hover at the target.
Off-policy (Wietske)
On-policy (Mathew)
Key findings
- Still air is solved by all controllers. PID matches the best RL controllers on success and tracks within centimetres.
- Composite wind separates them. PID’s episode completion collapses to roughly 10–13%, with corrective thrust lagging the gusts.
- Best off-policy: SAC. Significantly ahead of TD3 and TAD3 under wind.
- Best on-policy: PPO. Highest fraction-in-tolerance (0.900 vs PID’s 0.274) and the tightest seed spread.
- Two open hypotheses. TAD3’s mean-aggregation benefit over TD3 and SCR-PPO’s dynamic-clipping benefit were both inconclusive.
- Real-time feasible. RL inference takes about 2.6–2.9 ms against a 4 ms budget; PID takes about 44 µs.
Reading the two charts side by side. Both studies use the same environment, reward, curriculum and wind model, but they trained for different step budgets (1.5 M vs 3.5 M) and each report evaluates its own PID and success criterion. Compare controllers within a report; the detailed tables are on the individual pages.
Scope
Limitations and future work
Limitations
- One simulated airframe (Crazyflie 2), one hover task and one wind family; no physical flight.
- PID comparisons under composite wind are underpowered because PID rarely completes an episode.
- Tight episode boundaries end flights early, which limits recovery and statistical power.
Future work
- More seeds, vehicles and disturbance profiles.
- Cascaded controllers with multiple control loops.
- Pre-registered replication of the dynamic-clipping ablation.
- Latency measured on embedded flight hardware.
Documents
Read the work
The research proposal was written jointly; literature reviews and final reports are individual. Everything is collected on one page.
-
PDF
Open
Research proposal
Joint document, 10 pages: “Beyond PID: Benchmarking Reinforcement Learning Controllers for Quadcopter Stabilisation”.