Introduction
The last entry on this blog is from October 2023. Almost three years. There’s no dramatic story behind that: there was work, shifting priorities, and above all a stretch where the field moved so fast that writing about it felt like photographing a moving train. Every draft I started was outdated before I finished it.
So I decided to come back the only way that actually convinces me: with something built, not with a promise to write again. This post is about that something — a RAG chatbot that answers questions about this very blog’s posts, in Spanish or English, always citing the post the answer came from. It runs fully self-hosted: no paid API, no runtime secrets, no GPU, and it lives as an ephemeral demo: it gets provisioned on request, runs for about 20 minutes, and tears itself down.
There’s a symmetry I like here: to start writing again, I built a machine that reads everything I’d already written.
And there’s something more honest worth saying up front: I built this project together with a coding assistant, across long sessions of decisions and measurements. That’s exactly the kind of work I want to write about now, so this post also doubles as a sample of what that process looks like when it’s documented well.
The constraints defined the design (not the stack)
Before the first line of code, I wrote the rules. In a project like this the constraints are the design; the stack is just the consequence:
- $0 in paid API costs. LLM and embeddings run locally. No per-token bills on a public demo.
- Self-contained. No runtime secrets, no external calls in production. Everything the demo needs is baked into the images.
- CPU only. No GPU. None of the “it works but costs like a training server” trap.
- Ephemeral, with abrupt teardown. ~20-minute sessions, 8 minutes idle, plus a kill switch. The container can die mid-stream: there can be no state worth losing.
- OSS stack, public images, and public data only (the posts are already on the internet).
Almost everything else fell out of that, on its own:
| Piece | Choice |
|---|---|
| App | FastAPI (Python 3.13) + vanilla UI, SSE streaming |
| LLM | Qwen2.5-0.5B-Instruct via Ollama, local in the container |
| Embeddings | paraphrase-multilingual-MiniLM-L12-v2 (384-d), local |
| Vector store | Postgres 16 + pgvector, ephemeral, seeded on boot |
| Corpus | 34 posts (28 EN + 6 ES) → 278 chunks of ~400 tokens |
Architecture
The demo is three containers in one pod, sharing localhost: the app (the only one with
ingress), Ollama, and Postgres/pgvector. The app carries no auth of its own — a gateway in front of
it handles that, and also terminates TLS.
Browser (session) ──HTTPS──► Gateway (TLS + auth)
│ internal ingress :8080
▼
FastAPI (app)
│
Ephemeral pod — 2 vCPU / 4 GiB, shared localhost
├─ embed query ──────► sentence-transformers (multilingual MiniLM-L12-v2)
├─ top-3 cosine ─────► pgvector / Postgres 16 (278 chunks / 34 posts)
└─ grounded prompt ──► Ollama (Qwen2.5-0.5B-Instruct)
│
▼ SSE tokens + citations
BrowserThe expensive part — fetching the posts, cleaning them, chunking them, and computing embeddings —
never happens at runtime. It happens in CI, and the result is a dump.sql baked into the
database image. When the demo boots, Postgres just loads the dump and the corpus is ready. That’s
the difference between a startup measured in seconds and one measured in minutes, and in a demo
that lives for 20 minutes, that difference is everything.
The per-message flow is classic RAG, no frills:
- The question gets embedded (384-d, normalized).
- Top-3 by cosine similarity in pgvector, filtered by a threshold.
- The prompt is assembled: grounded system prompt + retrieved context + history + question.
- Qwen generates in streaming, the app relays it token by token over SSE, and attaches the citations (the posts the chunks came from, deduplicated by URL).
The retrieval query is exactly as simple as it should be:
SELECT url, title, lang, published, chunk_index, content,
1 - (embedding <=> %(vec)s::vector) AS score
FROM chunks
ORDER BY embedding <=> %(vec)s::vector
LIMIT %(k)sAnd the rule that makes the chatbot a chatbot about my blog rather than a generic chatbot lives in the system prompt:
You are a grounded assistant that answers ONLY using the provided context,
which comes from Alexis Alulema's blog posts.
- Reply in the SAME language as the user's question (English or Spanish).
- Base every statement strictly on the context. Do NOT use outside knowledge.
- If the context does not contain the answer, say —in the user's language— that
you can only answer questions about Alexis Alulema's blog posts.
- Cite the sources you use by their titles.Except the real grounding isn’t done by the prompt — it’s done by the threshold. If no chunk clears the minimum similarity, the app doesn’t even call the LLM and returns a fixed, hand-written message in the detected language. (More on why that decision turned out to matter even more than I thought, later.) A 0.5B model given no context makes things up; the best defense against hallucination is never giving it the chance.
Ingestion: the unglamorous work
Posts are fetched from the site’s public sitemap (not the blog’s private repo), cleaned, chunked, and embedded. Cleaning was, as always, the least glamorous and most decisive part:
- The site is Astro: the real content lives in
.post-content; header and footer are navigation. - Icon ligatures (Material Symbols) leaked in as stray words —
beenheresitting in the middle of a paragraph — same with tables of contents and their anchors. - KaTeX duplicated every formula: MathML + the LaTeX annotation + the rendered version. Three copies of the same theorem competing inside the same chunk.
- Classic mojibake:
requestsguessed Latin-1 and handed back broken accented characters. Forcing UTF-8 and normalizing /zero-width characters closed that.
🎓 The detail I most enjoyed solving: paraphrase-multilingual-MiniLM-L12-v2 truncates input
at ~128 tokens, but my chunks are 400 tokens long. Embed the chunk directly and you silently lose
three-quarters of the text, with no warning. The fix was to split each chunk into sub-windows of
≤128 tokens and mean-pool their vectors: the chunk the LLM sees stays large, and the vector
representing it actually represents all of it.
The embeddings are multilingual, so retrieval is cross-lingual for free: a question in Spanish retrieves English chunks and vice versa. The first end-to-end test asked “What are activation functions?” and retrieved both correct posts — the English one and its Spanish version.
Calibrating the threshold: where I nearly fooled myself
The similarity threshold is the single most important parameter in the project: it decides when the bot answers and when it says “I can only talk about Alexis’s posts.” Eyeballing it is tempting and it’s a mistake, so I wrote a small harness: a battery of labeled questions inside the corpus (EN, ES, and cross-lingual) and outside the corpus, that measures each group’s score distribution and recommends a cutoff.
The first calibration was a dream: clean separation, in_min = 0.438 vs out_max = 0.205, a
0.233 gap, and zero misclassifications. Threshold set right in the middle: 0.32. Case
closed.
It wasn’t closed. With the demo already running, nine out-of-corpus questions were tested and
two slipped through: React’s useState and useEffect produced a real answer instead of a
refusal. Kubernetes, Vue, Angular, and Tailwind all refused correctly in that batch. Reproducing it
locally against the same database image, the diagnosis was uncomfortable:
| Out-of-corpus question | top-1 |
|---|---|
| Vue.js composition API | 0.492 |
React useState | 0.475 |
React useEffect | 0.458 |
| React hooks | 0.388 |
…and the actual floor inside the corpus — the question about my Levenshtein-in-JavaScript post — sits at 0.425. In other words: below Vue. There is no threshold that closes that leak without also refusing to answer a post I actually wrote.
The mistake wasn’t the threshold — it was the test battery. My negatives were obviously unrelated topics (France, pizza, Taylor Swift). The negatives that matter are the close ones — the ones that brush up against your domain without being in it. A post about JavaScript algorithms and a question about frontend frameworks live millimeters apart in embedding space, and with 400-token chunks that distance shrinks even further.
The decision was explicit, not automated: raise it to 0.42 (tightens the margin without rejecting any real topic) and accept the React/Vue leak as a known, documented limitation. The harness, optimizing for pure accuracy, recommended 0.577 — which would have rejected “How do transformers work internally?”, the single most central topic on the blog. Optimizing the harness’s metric at the expense of your most important content is exactly the kind of decision that has to be made by hand.
Performance on CPU: the prompt is your budget
Once the demo was deployed, the expected problem showed up: slowness. And a metric that on CPU matters more than any other: TTFT (time to first token) — the prompt’s prefill before the first word comes out. The platform has a hard ceiling of 2 vCPU / 4 GiB, so there was no hardware lever to pull: the work itself had to get smaller.
Three rounds, each measured separately so the effect could be attributed correctly:
| Round | Change | Measured result |
|---|---|---|
| Perf I | LLM 1.5B → 0.5B + max_output_tokens 512 → 256 | generation ~25-30 tok/s (~3×) |
| Perf II | TOP_K 5 → 3 | TTFT ~17.6 s → ~10.1 s |
| Perf III | CHUNK_TOKENS 600 → 400 (+ recalibrated threshold) | cold TTFT ~5.7 s |
The lesson is accounting, not magic: on CPU, the retrieved context is the largest, most variable part of the prompt. Going from 5 chunks of 600 tokens to 3 chunks of 400 took the prompt from ~1800 to ~1200 tokens, and TTFT dropped to a third of the original. Token-by-token streaming does the rest of the work — it doesn’t reduce latency, but it completely changes how latency is perceived.
⚠️ One detail that nearly threw my measurements off: repeating the same question came back in ~1 s. That’s Ollama’s KV cache, not an improvement. The measurements that count are cold, novel questions.
And a rule fell out of this: changing chunk size forces you to recalibrate the threshold. Smaller chunks are more focused, the similarity density shifts, and a cutoff that used to be correct stops being correct. Moving to 400 tokens narrowed the gap between “in” and “out” from 0.233 to 0.069. Less latency, less grounding margin: that was the trade-off, made with the numbers in plain view.
Two bugs that taught more than the happy path
”What topics does this blog cover?” → refusal
A trivial question, and the bot refused it. The cause was elegant in its obviousness: no chunk summarizes the corpus. Every chunk is a fragment of one specific post, and a meta question doesn’t resemble any single fragment closely enough. It wasn’t a badly set threshold — it was a corpus gap.
The fix was an ingestion problem, not a runtime one: generate a synthetic per-language summary document at build time that lists the topics, and let it flow through the normal pipeline (chunk → embed → dump). It gets retrieved like any other chunk, and its citation points to the blog’s index.
The first version listed titles, and it half-failed: a title doesn’t always name the
technology. So I went back to the site’s HTML, and there it was, exactly what I needed: every post
already publishes its own tags (machine-learning, python, transformers, c#, cqrs,
ddd…), hand-curated by me at publish time, and never scraped by the cleaner. Real metadata beats
any keyword-extraction heuristic I could have invented. That question now retrieves the summary
with a top-1 of 0.694, comfortably over the threshold, and it’s the first suggested chip in the UI:
showing the user the corpus’s real scope from the first second is also the best mitigation against
off-topic questions.
”The chat has no memory” → it actually did
The report was: follow-ups like “and in Python?” feel like they have no memory. The natural hypothesis is missing history. But the history was there, complete, end to end: the client keeps it and resends it every turn, and the prompt includes it turn by turn.
The bug was one step earlier. Retrieval only embedded the latest question. A context-dependent follow-up, on its own, doesn’t carry enough signal to clear the threshold → zero chunks → the app refuses without ever calling the LLM, ignoring the fact that the history actually held the context. The feeling of “it doesn’t remember” wasn’t the model forgetting — it was retrieval never giving it the chance to remember.
The fix was two independent changes: a 5-turn window on the client (history was growing unbounded and bloating the prompt I’d worked so hard to slim down), and a contextualized retry: if retrieval on the question alone comes back empty and there’s history, retry once by prepending the user’s last turn — only for the search embedding, never for what’s actually sent to the LLM. The normal path doesn’t change, so the threshold calibration remains valid for it.
I’ll take this as the best lesson of the project: in a RAG system, the symptom almost never points to the layer at fault. “The model doesn’t remember” was a retrieval problem. “The threshold is wrong” was a corpus problem. “It’s slow” was a prompt-size problem.
The first real user (or: how my sister broke the demo in five minutes)
Everything above, I tested myself — and I know exactly what I built, so without meaning to, I write the questions the system expects. So I asked my sister to try it. She’s not a developer, hadn’t read anything about the project, and walked into the chat the way anyone actually would: by greeting it.
Her entire session was two messages long. She found three things.
“Hola” → answered in English. The language detector was a keyword heuristic, and hola wasn’t
on any list: no accented characters to signal Spanish, no recognized word, the score sat at 0-0 and
fell back to the default, which was English. All of my own testing had been full questions — “How
do transformers work?” — dense with signal. A bare greeting, the single most common way a human
opens a chat, was exactly the no-signal case.
“Who is Alexis?” → the stream cut off mid-answer. It managed to write “Alexis Alulema is” and the connection died. She didn’t try again: she closed the chat. The cause was at startup: Ollama lazily loads the model’s weights on the first real inference, and my health check only verified that the server responded. Every TTFT measurement I’d been so proud of had been taken with the model already warm. Her question was the first genuine inference in that container, it paid the full load cost, and the gateway cut the connection on timeout before it could finish. The fix was to preload the model during startup, before reporting healthy: the container now takes a few extra seconds to come up, and no visitor ever pays that bill.
And that’s where the third thing came from — the one that taught me the most.
The temptation to fix it with a prompt
Testing after those fixes, I ran into the canned refusal myself: I asked “What topics do you know?”, it didn’t match any chunk, and I got the same dry, fixed sentence every time. It sounds like a robot. The obvious idea: instead of the fixed sentence, call the LLM with a different prompt asking it to kindly acknowledge it doesn’t have that information and invite the user to ask about real blog topics. More human, more conversational.
I implemented it, spun up the full stack, and tested it with questions that were genuinely out of scope:
| Question (no chunk retrieved) | Response |
|---|---|
| What is the capital of France? | ”The capital of France is Paris.” |
| What’s the best pizza in the world? | Confidently invented a recipe |
| Who won the 2022 World Cup? | ”…Brazil won” |
The prompt said, literally, “Do NOT use outside knowledge to answer the original question.” The model ignored it all three times. And on the third one, on top of breaking the rule, it lied: Argentina won.
That’s the lesson, and it’s the most important one I’m taking away from this project: a 0.5B model doesn’t reliably respect negative instructions, so a hard guarantee can’t live inside a prompt. “Grounded-only” is a promise I’m making to whoever uses the demo; if honoring it depends on the model having the discipline to hold back, that’s not a guarantee — it’s a hope. I reverted the change: when there’s no context, the LLM simply isn’t invoked — the response is deterministic, written by me, and yes, kinder than the original one (it acknowledges it doesn’t know and suggests real topics). It has no conversational variety. It never makes things up. That trade-off isn’t even close to a hard call.
And there’s one detail that feels like the summary of this whole section: none of these three problems could have been caught by my test suite. All 72 tests were passing green while the bot greeted people in the wrong language, hung on the first-ever visitor, and invented World Cup finals. It took someone who knew nothing about the system, and the real stack actually running, to surface any of it.
What I deliberately didn’t do
- No fine-tuning. The model is fixed; the only thing that gets re-indexed is the corpus. Publishing a new post doesn’t retrain anything.
- No server-side persisted conversations. History lives on the client, which is exactly why the demo can tolerate being killed mid-session.
- No runtime re-indexing. The corpus is fixed per image. It refreshes at build time, not at deploy time: a workflow re-ingests, regenerates the dump, and republishes the image; the next demo that gets provisioned already uses the fresh corpus.
- No auth of its own in the app — the gateway handles that.
And the honest limitation, already mentioned: questions about adjacent frontend frameworks (React, Vue) can slip through and produce a weak answer instead of a clean refusal. It’s measured, documented, and accepted — which is a very different situation from “we didn’t know.”
Conclusion
This project doesn’t invent anything: RAG is a well-known, well-documented architecture. What was actually interesting lived on the other side — in the constraints. A 0.5B LLM, two vCPUs, no paid API, and a twenty-minute lifecycle turn decisions that are normally comfortable (top-k? chunk size? threshold?) into measurable trade-offs with visible consequences. And measuring each one separately, instead of changing three things at once, is what made it possible to attribute each improvement to its actual cause.
There’s an ending I genuinely like: this site’s publishing pipeline triggers corpus re-indexing. As soon as this post goes live, the chatbot will be able to answer questions about it — including, I imagine, the question of why it took me three years to start writing again.
The chatbot is available as a live demo, alongside the rest of my experiments, at alexisalulema.com/projects — it provisions on request, runs for a few minutes, and tears itself down. And the code is public, complete with its full log of decisions, measurements, and mistakes: github.com/alulema/rag-blogposts.
I’m back. And this time I’m bringing things I actually built, to talk about.