Blog

Notes on software development, machine learning, and engineering.

18 min read

What Do We Do When the Agent Doesn't Learn? RL Debugging Explained from Scratch

Part 8 of the Reinforcement Learning from scratch series with Agentic Racing. How to turn "the agent doesn't learn" into testable hypotheses, told through the real debugging of the project's RL pilot: at least six plumbing bugs before the first run, eight PPO runs with the lap fraction stuck, a baseline that had a bug of its own, misleading averages, termination versus truncation in ML-Agents, and the process mistakes that cost the most time.

reinforcement-learningmachine-learningagentsunity
19 min read

What Should an Agent Be Able to See? Observation Design Explained from Scratch

Part 7 of the Reinforcement Learning from scratch series with Agentic Racing. How you decide what information an agent gets, going one by one through the project pilot's 42 real observations: scaling, anticipation, memory, what can let a policy memorize a circuit, what it doesn't see (the rivals), the bugs that sent empty observations, and why the pilot, the expert, and the LLM strategist deliberately see different things.

reinforcement-learningmachine-learningagentsunity
26 min read

How Do We Tell an Agent What We Want? Reward Engineering Explained from Scratch

Part 6 of the Reinforcement Learning from scratch series with Agentic Racing. How a human goal gets translated into a reward, using the project's real RL pilot reward as a case study: scales, units, dense and sparse rewards, terminations that act as rewards, reward shaping, and why seven variants of the reward couldn't fix a problem that wasn't about the reward.

reinforcement-learningmachine-learningagentsunity
20 min read

Can We Combine Imitation and RL? BC, GAIL, and PPO as One System

Part 5 of the Reinforcement Learning from scratch series with Agentic Racing. How to chain Behavioral Cloning, GAIL, and PPO: what ML-Agents actually does when you combine them, how a two-run fine-tuning would work, why in this project the task reward could erase exactly what the demo needs, and how to design an experiment that can show the combination doesn't work.

reinforcement-learningmachine-learningagentsunity
27 min read

Can We Learn to Behave Like an Expert? GAIL Explained from Scratch

Part 4 of the Reinforcement Learning from scratch series with Agentic Racing. What GAIL is, how a discriminator turns imitation into a learned reward, exactly what ML-Agents does with it, and what would have happened if the project had actually used the GAIL configuration it prepared for its pilot.

reinforcement-learningmachine-learningagentsunity
26 min read

Can We Teach by Imitation? Behavioral Cloning Explained from Scratch

Part 3 of the Reinforcement Learning from scratch series with Agentic Racing. What Behavioral Cloning is, how ML-Agents implements it, why a small mistake turns into a big one, and what happened when the project set everything up to imitate its heuristic pilot without the log ever recording a single demonstration.

reinforcement-learningmachine-learningagentsunity
21 min read

How Do We Improve a Policy Without Breaking It? PPO Explained from Scratch

Part 2 of the Reinforcement Learning from scratch series with Agentic Racing. We open up PPO: policy gradient, the probability ratio, clipping, actor-critic and GAE, each piece tied to the project's real ML-Agents configuration and to what happened when PPO optimized exactly what we asked it to.

reinforcement-learningmachine-learningagentsunity
33 min read

How Does an Agent Learn? Teaching a Car to Drive

The first article in a series on Reinforcement Learning from scratch, using Agentic Racing as a lab: state, observation, action, policy, reward, return, value, Q and advantage, explained with the project's real RL pilot code, its numbers, and the failures that taught more than any equation.

reinforcement-learningmachine-learningagentsunity
12 min read

A Trip Planner That Reasons Without Generating: Chain-of-Agents with a 1.5B Model

I took the Chain-of-Agents pattern from chapter 7 of 30 Agents Every AI Engineer Must Build and turned it into a real demo: a local 1.5B model that writes only what it must, a JEV-style decision engine that answers with probabilities instead of text, and a budget-conflict loop that resolves without generating a single token. What worked, what the model made up, and how the code ended up writing the honest part.

agentsllmlanggraphollamapythonfastapi
18 min read

Agentic Racing: A Pilot That Never Learned to Drive, a Team Boss That Did Learn to Reason

I built a racing demo where a reactive pilot obeys a team-boss LLM that reasons per event. Eight attempts at training the pilot with reinforcement learning didn't work — the track was broken before the algorithm ever had a chance — and measuring how much the strategist actually contributes turned into the more interesting experiment: seven runs, real and partially fixable reasoning biases, and a lesson about trusting the full benchmark over the quick desk-check.

reinforcement-learningllmunityagentsollamapython
16 min read

Twenty Minutes of Infrastructure (and Then Nothing Left)

The demo behind each project card on my site doesn't exist until you ask for it, and stops existing twenty minutes later. This is the machinery behind that button: an ephemeral environment per session, a subdomain pool, a gateway, a reaper that had to be rebuilt, and the bugs that only showed up in production.

infrastructureazurecontainersgithub-actionscloudflaredevopsiac
17 min read

A RAG Chatbot Over My Own Blog (and the Comeback After Three Years)

I came back to writing by building something: a bilingual RAG chatbot over this blog's posts, 100% local, $0 in API costs, deployed as an ephemeral demo. Architecture, threshold calibration, CPU performance, and the bugs that taught more than the happy path.

ragllmpythonfastapipgvectorollamaembeddingsdocker
23 min read

Advanced Asyncio Topics: Beyond the Basics

Introduction Hello, asyncio enthusiasts! If you've been following along, you'll remember that we recently dived into the basics of Python's asyncio library in our previous blog post. We explored the f…

asynchronous-programmingpython
12 min read

'asyncio' tutorial for the programmer in a hurry

Introduction In the evolving world of Python programming, understanding the asyncio library is becoming increasingly important. asyncio allows for asynchronous, concurrent, and parallel programming in…

asynchronous-programmingprogrammingpython
15 min read

Transformers: how do they work internally?

Table of Contents Introduction Input Embedding Positional Encoding (PE) The Encoder Self-attention Mecanism Multi-head attention mechanism Feedforward Network The Decoder Masked Multi-head Attention M…

machine-learningpythontransformers
8 min read

Python Profiling - Memory Profiling (Part 3, Final)

Table of Contents memory_profilerPySpyDISassemblingFinal Recommendations memory_profiler Similar to line_profiler, memory_profiler provides detailed memory usage measurements, with the aim of efficien…

profilingpython
10 min read

Python Profiling - cProfile and line_profiler Tools (Part 2)

Table of Contents cProfileSnakeViz, for cProfile insightsLine-by-line ProfilingAbout @profile decoratorOther useful toolsgprof2dotPyinstrumentConclusionAppendixInstall with pip or conda cProfile The P…

profilingpython
5 min read

Python Profiling - Time Profiling (Part 1)

Table of Contents IntroductionTime Profiling Toolsfunctools.wrapstimeit moduletime / gtimeConclusion Introduction Many times, the code we write requires optimizations, and profiling helps us find the …

profilingpython
3 min read

JavaScript Levenshtein Algorithm Implementation

One of the most known algorithms used by spell checkers and auto-complete fields is perhaps the Levenshtein algorithm, which basically minimizes the Levenshtein distance between the target word and th…

javascriptlevenshtein
12 min read

Support Vector Machines (SVM) for Classification

The purpose of this document is to present the linear classification algorithm SVM. The development of this concept has been based on previous ideas that have supported the development of SVM as an al…

matlabmlsvm
1 min read

DDD Clean Architecture Template

(GitHub Repo: https://github.com/alulema/DDD-CleanArchitectureTemplate) These last 10 months I’ve been delighted working with ASP.NET Core, considering the improvements made by Microsoft to this frame…

c#cqrsddddesign-patternsmediator
3 min read

C# Sudoku Solver

(GitHub Repo: https://github.com/alulema/SudokuSolverNet) I was revisiting a couple of basic AI concepts:  Depth First Search and Constraint Propagation, and I found a very good explanation …

aic#constraint-propagationdepth-first-search
2 min read

TensorFlow High-Level Libraries: TF Estimator

TensorFlow has several high-level libraries allowing us to reduce time modeling all with core code. TF Estimator makes it simple to create and train models for training, evaluating, predicting and exp…

machine-learningpythontensorflow
3 min read

TensorFlow Way for Linear Regression

In my two previous posts, we saw how we can perform Linear Regression using TensorFlow, but I’ve used Linear Least Squares Regression and Cholesky Decomposition, both them use matrices to resolve regr…

linear-regressionmachine-learningpythontensorflow
4 min read

Cholesky Decomposition for Linear Regression with TensorFlow

beenhereUpdate 10.2023This entry has been updated with versions TensorFlow 2.12.1 and PyTorch 2.0.1 Several years have already passed since the onset of the Deep Learning boom. I have witnessed impres…

choleskylinear-regressionmachine-learningpythonpytorchtensorflow
2 min read

Linear Least Squares Regression with TensorFlow

Linear Least Squares Regression is by far the most widely used regression method, and it is suitable for most cases when data behavior is linear. By definition, a line is defined by the following equa…

linear-regressionmachine-learningpythontensorflow
2 min read

Classification Loss Functions (Part II)

In my previous post, I mentioned 3 loss functions, which are mostly intended to be used in Regression models. This time, I’m going to talk about Classification Loss Functions, which are going to be us…

machine-learningpythontensorflow
4 min read

Loss Functions (Part 1)

Implementing Loss Functions is very important to machine learning algorithms because we can measure the error from the predicted outputs to the target values. Algorithms get optimized by evaluating ou…

machine-learningpythontensorflow
13 min read

Activation Functions (updated)

beenhereUpdate 09.2022This entry has been updated to TensorFlow v2.10.0 and PyTorch 1.12.1 Table of Contents What is an activation function?Activation FunctionsSigmoidReLU (Rectified Linear Unit)ReLU6…

machine-learningpythontensorflow
3 min read

Working with Matrices in TensorFlow

Matrices are the basic elements we use to interchange data through computational graphs. In general terms, a tensor can de defined as a matrix, so you can refer to Declaring tensors in TensorFlow in o…

machine-learningpythontensorflow
2 min read

Understanding Variables and Placeholders in TensorFlow

Usually, when we start using TensorFlow, it’s very common to think that defining variables is just as trivial as a HelloWorld program, but understanding how variables (and placeholders) work under the…

machine-learningpythontensorflow
4 min read

Declaring tensors in TensorFlow

[Requirement: Tensorflow and NumPy installed on Python +3.5][Requirement: import tensorflow as tf][Requirement: import numpy as np] Tensors are the primary data structure we use in TensorFlow, and, as…

machine-learningpythontensorflow
5 min read

Java Streams API in brief

First, let’s define what a stream is in Java 8: a sequence of functions, actions, inputs, and outputs (better defined as a “pipeline”). Streams API provides functional-style operations to transform th…

data-engineeringjava
6 min read

Does it worth to work as a freelancer?

BACKGROUND Usually many of us graduates from college have dreamed of working in a large company with great salary, prestige and traveling around the world; or maybe we’ve had the “million-dollar idea”…

lifestyle
1 min read

Cross-Platform Communications: gRPC Server and Client: Node.js

This is my second video on gRPC, this time I was experiencing Node.js to make an app communicate with a gRPC .NET Core app. Here the details about Node.js implementation: https://youtu.be/3wa0bs_Jw6M…

grpcjavascriptvscode
1 min read

Installing Redis on Azure Ubuntu VM

In one of my latest developments I needed to have a local Redis for caching requirements. I feel comfortable on terminal/command line, so it wasn’t hard to have an Ubuntu VM on VirtualBox, and it is v…

redisubuntu
1 min read

ASP.NET Core Basics – Part 2

Now, we continue with the series ASP.NET Core Basics. At this time we are been focused in MVC specific topics, which will be covered in a 7 mins video. Contents: MVC. ControllersMVC. Models …

c#dotnetvscode
1 min read

Installing SQL Server for Linux on Docker Container

It is very common requiring MS SqlServer if you are developing ASP.NET web applications; because of this, having a local Sql Server is a great plus if you are developing web apps on ASP.NET Core on Ma…

macosxsqlservervscode
1 min read

ASP.NET Core Basics – Part 1

I’m starting this blog with a series of videos focused on ASP.NET Core. This tutorial has been completely developed on C# + Visual Studio Code + Mac OSX, but (almost) all commands and concepts are ful…

c#dotnetvscode