Series: Reinforcement Learning from scratch, with Agentic Racing as the lab
- Part 1 · How Does an Agent Learn? Teaching a Car to Drive
- Part 2 · How Do We Improve a Policy Without Breaking It? PPO Explained from Scratch
- Part 3 · Can We Teach by Imitation? Behavioral Cloning Explained from Scratch
- Part 4 · Can We Learn to Behave Like an Expert? GAIL Explained from Scratch
- Part 5 · Can We Combine Imitation and RL? BC, GAIL, and PPO as One System
- Part 6 · How Do We Tell an Agent What We Want? Reward Engineering Explained from Scratch
- Part 7 · What Should an Agent Be Able to See? Observation Design Explained from Scratch
- Part 8 · What Do We Do When the Agent Doesn’t Learn? RL Debugging Explained from Scratch (this article)
Where we left off
The seven earlier parts each told, from their own angle, why Agentic Racing’s RL pilot never learned to complete laps consistently: the reward had shortcuts (part 6), the observations didn’t see corners in time (part 7), PPO optimized the wrong objective very well (part 2), and the imitation that could have helped was never run (parts 3 to 5).
This part puts all of that together around the question the project asked itself for a week in September:
The agent doesn’t learn. Why?
The short answer is that the question was badly framed. “It doesn’t learn” isn’t a symptom: it’s a conclusion. Debugging an RL system means turning it into hypotheses that can be checked one by one. This article tells how the project did it, what worked, and which process mistakes cost more than any bug.
1. “It doesn’t learn” isn’t a symptom
Very different situations hide behind that phrase. Agentic Racing went through almost all of them:
| What you see | In Agentic Racing |
|---|---|
| not a single episode completes | 2 million steps with zero completed episodes: the car never moved |
| the reward doesn’t change | the first run converged at 1.85 million steps and stayed flat for 18 million more |
| the reward rises, the behavior doesn’t | the seventh run: reward +35%, completed laps from 2% to 3% |
| it improves and then gets worse | the third run climbed to ~10; the fourth fell back to ~4.8 |
| it learns something, just not what we wanted | the fourth run learned to stop on purpose |
Each row calls for a different investigation. The first is a plumbing bug; the second can be a local optimum; the third, a reward that measures something else; the fourth, variance; the fifth, a shortcut. Treating them all as “the agent doesn’t learn” leads to trying fixes at random.
2. The system you have to debug
An RL agent doesn’t learn in a vacuum. Between the idea and the learning there’s a long chain: the simulator, its connection to Python, the sensors, turning decisions into actions, the physics, the reward, how episodes end, the evaluation, and the baseline you compare against. Any broken link can produce the same symptom.
This is Agentic Racing’s map, with the real bugs from its devlog on each link:
What stands out most on the map is what isn’t there: the algorithm. All eight runs used PPO, and none of those bugs was fixed by touching it. Part 2 had already shown it with the table in section 9: what changed between runs was the reward, the agent’s perception, or the physics.
3. First, make the plumbing work
Before the first run could train, the project spent three days debugging things that had nothing to do
with learning to drive. These are six of them (there were more: several in the player build itself, and
a Python environment outside the versions mlagents pins):
- The headless player crashed while registering the agent. ML-Agents only reported an exit code; the real log showed up when running the executable by hand. The fix was a software framebuffer instead of the no-graphics mode.
- The native gRPC library wasn’t where it was being looked for, and the player decided there was no trainer and switched to inference mode, with no visible error.
- ML-Agents’ communicator doesn’t work when compiled with IL2CPP. That forced moving the training player from Linux to Windows with Mono.
- The observation size didn’t match: 12 had been agreed and 27 arrived (part 7).
- The environment hung at ~63,000 steps. With the observations fixed, training ran until Unity throttled the simulation when the window lost focus, and the trainer gave up waiting.
- Two million steps, zero episodes. Decisions were requested but the action was never executed (and the observations arrived empty): the car didn’t move. It was the order in which the car’s components were assembled.
The pattern that worked kept repeating: find the log that tells the truth (the player’s, not Python’s), add targeted logging where the hypothesis called for it and, when needed, reproduce the failure on another machine to rule out the environment. And the first run was only launched after a short test confirmed there were episodes, actions, and complete observations.
The lesson is the cheapest of the whole series: “0 episodes completed” and “observations padded with zeros” were written in the logs. A training run that runs isn’t a training run that learns, and the first thing to verify is that the experience it receives is the one we think it is.
4. A baseline you can believe
The project’s most useful diagnostic tool wasn’t a PPO metric: it was a hand-written controller in the same environment. If it can’t complete the lap, the problem isn’t the learning.
The project built it late, after the fifth run, and used it badly at first. The first evaluation had an inverted sign in the controller itself; the second showed cars dying anywhere, always stuck against a wall. After the car was given steering at low speed to get off the wall, one episode reached 82% of the lap, and that was read as “the environment is learnable, the problem is the RL”. The other half of the data, that on average the controller also got stuck, turned out to be the important one (part 6).
And the baseline had bugs of its own. One was only found in October, while writing part 7 of this series: the timer that decides when the heuristic pilot reverses off a wall counted five times too slowly, because it added one physics step per call and the function is only called once per decision. Its threshold, now 1 second, amounted to 5 real seconds. It’s fixed now, but it was there from the sixth run on, and also in the mixed-field experiments (section 10).
A baseline is code too, and it needs debugging too.
5. Metrics that aren’t the reward, and don’t average too much
Part 6 told why the project built an independent evaluator after the fourth run: the reward didn’t say the policy never braked, and the evaluator did (mean brake of 0.02).
But an evaluator can lie too, and the project lived through that twice:
- A mean that described no car. In one evaluation of the heuristic pilot, the report gave a mean speed of 5.3 m/s. The per-car log, once a second, showed something else: two or three cars were driving at about 20 m/s and the other six or seven were stuck near the start, in a “sawtooth”. The mean looked like neither group.
- Episodes that didn’t exist. In another evaluation, the count of episodes ended by the time limit was inflated by spurious resets of the evaluator itself while loading the model.
The rule that comes out of it: before trusting a summary, look at a few individual cases. A log of one line per car per second solved in one evaluation what several aggregate reports hadn’t shown.
6. The action isn’t the effect
If the policy asks to turn and the car brakes, the problem isn’t in the policy. In Agentic Racing, the lateral grip model erased the car’s lateral velocity instead of redirecting it: any hard turn was a sudden stop (part 1). The devlog read that as the reason both the agent and the heuristic pilot didn’t turn hard: in that physics, turning cost speed.
The key tool was logging what the car did, not just what the policy asked for: a trajectory dump at the end of each episode showed cars with nearly constant steering, zero speed, and sliding sideways; the car “spins and slides sideways”, says the devlog. No hyperparameter would have shown that. But the dump showed the symptom, not the cause: the same data was first read as proof that the walls were failing, then as a spawn-position problem, and only when the code’s arithmetic was gone over (each step removed 18% of the lateral velocity) did the grip show up. And fixing the grip barely moved the eighth run: the devlog ended up blaming the paralysis on the procedural tracks, not on a deep physics bug (part 6). Seeing clearly what happens is necessary; it isn’t enough to know why it happens.
7. Ending or interrupting: how the episode ends matters
An episode can end because the task ended (the car completed the lap or left the track) or because time ran out. They’re not the same, and ML-Agents treats them differently:
- With
EndEpisode(), the episode is terminal: the next state’s value is taken as zero, because there’s no future left to collect. - On reaching
MaxStep, the episode is interrupted: the trainer uses the critic’s estimate for the next state’s value, because the task didn’t end, only the clock ran out.
In Agentic Racing, the completed lap and every failure ending (off track, stopped, wrong way, no
progress) use EndEpisode(), and the time limit uses MaxStep. That distinction is done right. But the
project had a truncation bug of another kind: the limit was 4,000 steps (80 seconds), and a lap of the
fixed circuit takes about 95. While the limit stayed there, at the heuristic pilot’s pace back then no
episode on that circuit could end in a completed lap; the heuristic pilot reached 86% and ran out of time. It was raised to 6,000 steps.
And part 6 showed the other side: which conditions end the episode, and what they cost, is part of the reward design. The fourth run’s shortcut came from exactly there.
8. When to look at PPO
Only once the experience is coherent does it make sense to look at the algorithm. In Agentic Racing, PPO’s hyperparameters barely changed: runs were shortened, the entropy coefficient went from 0.005 to 0.01 to explore more, and little else. Part 2 showed that PPO did its job: it optimized, stably, exactly the objective we gave it.
The training metrics did help, but read as signals about the process, not as a verdict:
| Metric | What it told the project |
|---|---|
| cumulative reward | where each run plateaued; not whether the car completed laps |
| episode length | whether the agent survived longer or died sooner (in the first run, it dropped while the reward stayed flat) |
| critic loss | whether the critic understood the world: it rose in almost every run without curvature observations and settled in the seventh |
| entropy | whether the policy was still exploring or had already made up its mind |
None of them ever said “the grip brakes the car” or “the tracks have impossible corners”. Those answers came from looking at the behavior.
9. Exploration: when the agent never sees what it needs to learn
In the last runs, between 91 and 98% of episodes ended off the track about 200–250 metres in, at the first real corner. The hypothesis at the time was about exploration: the agent reached the corner again and again, but almost never stumbled onto the brake–turn–accelerate sequence by chance, so it almost never lived through the experience that would have taught it braking was worth it. (Later, the grip and the procedural tracks came to explain a good part of that.)
There are three classic ways out, and the project had all of them on the table:
- Curriculum learning: start with gentle corners and make them harder. It was noted as the next step; it was never done.
- Imitation: start from the expert’s demonstrations (parts 3 to 5). It was configured; it was never run.
- Make the problem easier: the fixed circuit, with 120 m radius corners. That’s what unblocked the heuristic pilot, but the RL pilot was never retrained there.
10. One run isn’t evidence
Every run in the project was a single run: none was repeated with another seed, and the configuration doesn’t fix one. The third run showed what that costs. It climbed to a reward of ~10, with episodes five times longer, and looked like the hoped-for breakthrough. The fourth, with the same configuration and more steps, fell back into the pit. The devlog summed it up as “favorable variance from one run”.
The same project got this right later. In the experiment pitting the LLM strategist against the heuristic one, it ran series of 18 races with the grid rotated, reported the mean with its standard deviation, and the first result went against expectations. The difference between one phase and the other wasn’t tools: it was discipline.
11. The process mistakes that cost the most
Bugs get fixed. What cost the project the most time was its own habits while debugging:
- Changing several things at once. One iteration changed the steering physics, the edge reward, the wrong-way condition, and the heuristic pilot together. The seventh run changed observations and reward together (part 7). In both cases, any improvement was impossible to attribute.
- Declaring the cause before confirming it. Three devlog entries, written within a few hours, are titled “root cause”: the 12 m corners, the zero-thickness walls (“at last”), and the lateral grip (“for real this time”). The walls were ruled out later: three different versions gave the same result.
- Reading the baseline in favor of the hypothesis. One controller episode at 82% was taken as “the environment works”, when the average said the opposite (section 4).
- Training long before knowing how long was needed. The first run lasted 20 million steps and converged at 1.85: about 90% of the compute added nothing.
- Iterating on the reward when the problem wasn’t the reward. Seven variants, and the lap fraction didn’t move (part 6).
In the project’s defense, it also did things right: a short test before the first long run, a devlog that recorded every hypothesis with its result (without it this article wouldn’t exist), and an honest final decision. After the eighth run, instead of iterating further, it asked what the demo really needed, and the answer wasn’t an RL pilot (part 5).
12. A debugging sequence, applied
Putting it all together, this is the sequence the project would have liked to follow from the start, and what happened with each step:
| Step | What to do | In Agentic Racing |
|---|---|---|
| 1 | Describe the symptom with evidence | yes: the devlog records each run with its numbers |
| 2 | Verify the plumbing without training | yes, by necessity: at least six bugs before the first run |
| 3 | Inspect real observations and actions, and their effect | late: the trajectory dump arrived after the seventh run, in the heuristic evaluations |
| 4 | Validate the reward with known cases | partly: the shortcut was explained with numbers, but the expert’s reward was never measured |
| 5 | Verify terminations and truncations | well designed, with a time limit that was too short |
| 6 | Have a simple baseline | late, misread at first, and with a bug of its own |
| 7 | Train short before training long | the short test, yes; the first run, too long |
| 8 | Change one thing per experiment | no, several times |
| 9 | Evaluate with independent metrics, across several seeds | metrics yes, from the fourth run on, on nine tracks; training seeds, never |
| 10 | Document the result, negative ones too | yes: it’s the best part of the project |
13. Symptoms and hypotheses, with the project’s cases
| Symptom | Hypothesis to rule out first | Agentic Racing’s case |
|---|---|---|
| no episode ever ends | the decision and action plumbing | the actions never reached the car |
| reward flat from very early on | local optimum, reward with no useful signal | first and second runs |
| stops on purpose | ending the episode pays more than continuing | fourth run |
| drives fast and goes off | can’t see the corner coming, or braking doesn’t pay | fifth run; the curvature observation arrived in the seventh |
| won’t turn hard | the physics turns a turn into braking | the lateral grip (fixed, without moving the eighth run) |
| the reference controller can’t either | the environment can’t be solved | the procedural tracks |
| reward up, behavior not | the reward measures something else | seventh run |
| a very good result that doesn’t repeat | variance between runs | third run |
This table doesn’t diagnose on its own: every symptom has several possible causes. What it does do is tell you where to start looking, which is half the job.
Conclusion
Debugging an RL system isn’t finding the right combination of hyperparameters. It’s turning an opaque training run into a system you can observe, test, and explain, link by link.
Agentic Racing leaves a concrete version of that idea:
- verify the plumbing before blaming the learning: logs often tell the truth before curves do;
- have a hand-written baseline, and debug it too;
- look at individual cases before believing an average;
- log what the car does, not just what the policy asks for;
- tell ending apart from interrupting, and check that the limits make sense;
- change one thing at a time, and don’t declare the cause until the evidence confirms it;
- and document everything, including what didn’t work.
The goal of debugging isn’t for the curve to go up. It’s to understand why it doesn’t.
If you landed directly on this article, the seven earlier parts are linked at the top, and Agentic Racing: A Pilot That Never Learned to Drive tells the project’s full story.
References
- Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms.
- Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction, 2nd edition. MIT Press.
- Henderson, P. et al. (2018). Deep Reinforcement Learning that Matters. AAAI.
- Agarwal, R. et al. (2021). Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS.
- OpenAI. Spinning Up: Proximal Policy Optimization.
- Unity Technologies. Unity ML-Agents Toolkit,
mlagents1.1.0 release: the handling of interrupted episodes inml-agents/mlagents/trainers/ppo/trainer.pyandml-agents/mlagents/trainers/optimizer/torch_optimizer.py, andHeuristicPolicyincom.unity.ml-agents4.0. - The full debugging devlog (
docs/Devlog.md), the evaluator, and the code are at github.com/alulema/agentic-racing.