Individual contribution · On-policy RL
Beyond PID: Benchmarking Reinforcement Learning Controllers for Quadcopter Stabilisation
Mathew Maartens · Supervisor: Krupa Prag · University of Cape Town
A2CPPOSCR-PPOPID
Summary
What this report did and found
This report benchmarks three on-policy RL controllers, Advantage Actor–Critic (A2C), Proximal Policy Optimisation (PPO) and Stabilising Constraint Regularisation PPO (SCR-PPO), against a tuned PID baseline for quadcopter hover. All controllers share one environment, reward, disturbance specification and evaluation protocol; the RL controllers also share the training curriculum and step budget.
Every controller solved still air at about 0.95 fraction-in-tolerance (FIT). Composite wind separated them: PPO 0.900, SCR-PPO 0.796, A2C 0.458, PID 0.274. A 30-seed matched ablation favoured dynamic over fixed clipping (0.861 vs 0.813 FIT) but did not pass the Holm-corrected decision rule (p = 0.084).
Goals
Research questions
Method
The controllers and the ablation
A2C → PPO → SCR-PPO
- A2C: synchronous actor-critic, one update per rollout.
- PPO: clipped surrogate objective allows several minibatch epochs per rollout, so samples are reused.
- SCR-PPO: replaces PPO’s fixed clip ε with a per-minibatch value driven by the spread of actions, bounded between εmin and εmax.
Flat PID baseline
A non-cascade PID: position and attitude loops computed in parallel and combined before motor mixing, with derivative terms from measured velocities. Gains were selected through a separate 100-trial search under the same evaluation conditions.
Training and validation
- 100-trial Optuna TPE search per algorithm (2 M steps), ranked by composite FIT minus its standard error.
- 5 shortlisted configurations × 5 seeds (2.5 M steps), then 10 benchmark seeds at 3.5 M steps.
- Curriculum difficulty ramps to 1.0 over 85% of the budget; 30% of episodes are floor-level “anchor” spawns.
Dynamic-clipping ablation
SCR-PPO versus an otherwise identical arm with fixed ε = 0.1761, the mean of the dynamic values measured in two instrumented runs. 30 matched seeds (200–229) per condition; Holm-corrected Wilcoxon on final FIT.
Disclosed deviations. A2C’s selected configuration was chosen by a documented post-hoc override because all candidates had zero composite success, and the ablation was expanded from 10 to 30 seeds after inspecting initial results.
Selected RL configurations
| Method | LR | Worlds | Steps/world | Rollout | Batch | γ | GAE λ | Layers |
|---|---|---|---|---|---|---|---|---|
| A2C | .003826 | 64 | 10 | 640 | 640 | .98719 | .96404 | 64, 64 |
| PPO | .001521 | 512 | 32 | 16384 | 2048 | .98741 | .95403 | 128, 128 |
| SCR-PPO | .001111 | 1024 | 16 | 16384 | 1024 | .98349 | .93089 | 64, 64 |
Results
Final controller performance
| Metric | PID | A2C | PPO | SCR-PPO |
|---|---|---|---|---|
| Still air | ||||
| FIT ↑ | 0.954 (0.000) | 0.954 (0.001) | 0.952 (0.003) | 0.952 (0.002) |
| Success ↑ | 1.000 | 1.000 | 1.000 | 1.000 |
| Completion ↑ | 1.000 | 1.000 | 1.000 | 1.000 |
| RMSE (m) ↓ | 0.0236 | 0.0345 | 0.0171 | 0.0114 |
| Composite disturbance | ||||
| FIT ↑ | 0.274 (0.038) | 0.458 (0.317) | 0.900 (0.033) | 0.796 (0.268) |
| Success ↑ | 0.000 | 0.300 | 0.700 | 0.633 |
| Completion ↑ | 0.100 | 0.433 | 0.733 | 0.633 |
| RMSE (m) ↓ | 0.7653 | 0.5897 | 0.1695 | 0.2741 |
Still air. All controllers succeeded and completed every episode. After Holm correction, SCR-PPO had significantly lower final-window RMSE than A2C and PID, and PPO beat A2C.
Composite wind. PPO led on FIT, success, completion and all-episode RMSE. In the all-episode FIT family, PPO beat A2C and PID, and SCR-PPO beat PID, after Holm correction; PPO and SCR-PPO were not statistically separated. PID comparisons on completion-conditioned RMSE kept only 2–3 pairs and are underpowered.
Time to arrival
| Controller | Still air | Composite |
|---|---|---|
| Flat PID | 0.927 (0.005), n = 10 | 19.340 (0.240), n = 2 |
| A2C | 0.920 (0.030), n = 10 | 9.520 (7.940), n = 8 |
| PPO | 0.947 (0.027), n = 10 | 1.117 (1.430), n = 10 |
| SCR-PPO | 0.940 (0.035), n = 10 | 1.097 (0.071), n = 8 |
Learning and reliability
How each algorithm learned
Normalised composite learning-curve area
Bars are scaled to the 0–1 range of the area.
Key observations
- Nominal hover is learned before disturbance rejection. A2C hit 0.84 still-air FIT by 400 K steps but plateaued under wind.
- PPO was the most sample-efficient and reliable (composite FIT SD 0.033).
- SCR-PPO was strong but seed-sensitive (SD 0.268; seeds 201 and 206 fell sharply).
- The curriculum is coupled to training progress, so curriculum effects cannot be isolated causally.
RQ3
Does dynamic clipping help?
| Condition | Dynamic FIT | Fixed FIT | Mean diff. [95% CI] | rrb | p (unadjusted) | Significant |
|---|---|---|---|---|---|---|
| Still air | 0.9531 | 0.9536 | −0.0006 [−0.0012, −0.0001] | −0.415 | 0.047 | No |
| Composite | 0.8613 | 0.8130 | +0.0483 [−0.0243, 0.1198] | +0.363 | 0.084 | No |
Under composite wind dynamic clipping won on 19 of 30 paired seeds, and composite success was 0.744 vs 0.589 (supporting, not part of the corrected family). Instrumentation confirmed that εt really varied during training (means 0.1829 and 0.1693, range about 0.08–0.29, never clamped), but the mechanism behind the higher point estimate remains unexplained. The result is promising but unconfirmed and warrants a pre-registered replication.
Media
Flights
Discussion
Limitations and future research
Limitations
- The PID baseline is flat, not cascaded; results do not extend to stronger classical controllers.
- One simulated vehicle, task, disturbance family and training budget.
- A2C’s selected setting sits on the lower boundary of the searched world count.
- The ablation sample was expanded after inspecting initial results.
Future research
- Trajectory tracking, other vehicles, more disturbance profiles, and physical flight.
- Pre-registered clipping replication with more instrumented runs and several fixed clip values.