<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Engineering AI with Siva]]></title><description><![CDATA[Engineering AI with Siva]]></description><link>https://engineering-ai-with-sivahashnodedevcopyslug.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Wed, 18 Nov 2026 13:45:55 GMT</lastBuildDate><atom:link href="https://engineering-ai-with-sivahashnodedevcopyslug.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[I Built Cairnly: An Open-Source Platform to Learn AI Engineering and Explore Production-Ready RAG]]></title><description><![CDATA[Cairnly is a free AI engineering course with a citing tutor, plus the open-source RAG app behind it: Python, FastAPI, Ollama, pgvector and LLM evals.]]></description><link>https://engineering-ai-with-sivahashnodedevcopyslug.hashnode.dev/cairnly-open-source-ai-engineering-rag</link><guid isPermaLink="true">https://engineering-ai-with-sivahashnodedevcopyslug.hashnode.dev/cairnly-open-source-ai-engineering-rag</guid><category><![CDATA[RAG ]]></category><category><![CDATA[Artificial Intelligence]]></category><category><![CDATA[Open Source]]></category><category><![CDATA[Python]]></category><category><![CDATA[llm]]></category><dc:creator><![CDATA[Siva Cherukuri]]></dc:creator><pubDate>Tue, 17 Nov 2026 18:47:26 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6afc9c10d014da0e914077d7/d09e2a4a-14e1-4dfc-8aa6-dc4ee9929977.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><img src="https://raw.githubusercontent.com/CherukuriSiva/Cairnly/main/docs/images/ask-demo.gif" alt="Asking Cairnly a question: the answer types out with numbered citations and source cards" /></p>
<h2>Why I built Cairnly</h2>
<p>Building a RAG demo takes an afternoon. You split some documents, store embeddings, send the top matches to an LLM and get an answer back. It works on the first few questions you try.</p>
<p>The harder parts come after that. How do you know the right document was retrieved? What should the app do when the answer is not in your documents? How do you stop prompt injection without blocking honest users? How do you trace a bad answer, and how do you run all this without a big cloud bill?</p>
<p>I wanted one place where I could learn AI engineering <strong>and</strong> see all of these parts working together in a real app. So I built <strong>Cairnly</strong> and open-sourced it: a free AI engineering course with a tutor that cites its sources, plus the full code of the RAG app that runs it. That means local models, evals, safety checks, tracing and deployment, all in one repository.</p>
<ul>
<li><strong>Live site:</strong> <a href="https://cairnly-blond.vercel.app">cairnly-blond.vercel.app</a></li>
<li><strong>Try the tutor:</strong> <a href="https://cairnly-blond.vercel.app/ask">cairnly-blond.vercel.app/ask</a></li>
<li><strong>Recorded demo</strong> (works even when the GPU is asleep): <a href="https://cairnly-blond.vercel.app/ask?demo=1">cairnly-blond.vercel.app/ask?demo=1</a></li>
<li><strong>GitHub:</strong> <a href="https://github.com/CherukuriSiva/Cairnly">github.com/CherukuriSiva/Cairnly</a></li>
</ul>
<p>In this post I'll walk through what Cairnly is, how its RAG pipeline works, what the evaluation numbers show (including where it falls short), and how you can run it on your own machine.</p>
<h2>What is Cairnly?</h2>
<p>Cairnly has two sides:</p>
<table>
<thead>
<tr>
<th>Learn with Cairnly</th>
<th>Build with Cairnly</th>
</tr>
</thead>
<tbody><tr>
<td>126 free lessons in six tracks, 7 roadmaps and 29 insights</td>
<td>Hybrid search (keyword + vector) with a cross-encoder reranker</td>
</tr>
<tr>
<td>A tutor whose answers cite the exact lesson section</td>
<td>An eval suite of 627 questions, gated in CI</td>
</tr>
<tr>
<td>An honest "not in the lessons" instead of a guess</td>
<td>A confidence gate tuned on measured trade-offs</td>
</tr>
<tr>
<td>Quizzes, progress tracking and ⌘K search</td>
<td>An input guard and output check, with their misses published</td>
</tr>
</tbody></table>
<p>The code is MIT licensed. The lessons and docs are CC BY-NC-SA 4.0, so you can share and adapt them with credit, but not commercially.</p>
<h2>Learn with Cairnly: lessons, roadmaps and an AI tutor</h2>
<p><img src="https://raw.githubusercontent.com/CherukuriSiva/Cairnly/main/docs/images/home-dark.png" alt="The Cairnly home page in dark mode" /></p>
<p>The lessons are organized into six tracks: <strong>ML Foundations</strong>, <strong>AI Engineering</strong>, <strong>AI Systems &amp; MLOps</strong>, <strong>DSA &amp; Problem Solving</strong>, <strong>Full-Stack Development</strong> and a <strong>Career &amp; Portfolio Kit</strong>. Next to them are roadmaps that suggest an order, and short insights that summarize public posts with credit to their authors.</p>
<p>Every lesson is a plain MDX file in the <code>content/</code> folder, and a single <code>syllabus.yaml</code> sets the order. Because the content is public, anyone can fix a lesson with a pull request.</p>
<p>The tutor at <code>/ask</code> answers questions <strong>only from these lessons</strong>. Each answer has numbered citations like <code>[1]</code>, and each citation opens the exact lesson section it came from (<code>/&lt;track&gt;/&lt;lesson&gt;#&lt;section&gt;</code>). Every lesson page also has an "Ask Cairnly about this lesson" button that opens the chat with the search limited to that track.</p>
<p>If the lessons don't cover a question, the tutor says so instead of guessing:</p>
<blockquote>
<p>I couldn't find this in the knowledge base, so I won't guess. Try rephrasing, or ask about a topic the lessons cover.</p>
</blockquote>
<p>For a learning tool, a confident wrong answer with citations that don't support it is worse than no answer.</p>
<h2>Build with Cairnly: the open-source architecture</h2>
<p>Here is the full flow, from lesson files to a cited answer:</p>
<p><img src="https://cdn.hashnode.com/uploads/covers/6afc9c10d014da0e914077d7/be9c1cfe-1553-4b2a-8eae-49a3e92ce08f.png" alt="Cairnly RAG flow: lessons are chunked by heading and stored in Postgres; a question goes through the input guard, follow-up rewrite, hybrid search with RRF, cross-encoder reranking and a 0.66 confidence gate, then qwen3:8b on Ollama, an output check, and a streamed answer with numbered citations" /></p>
<p>A few design choices shaped everything else:</p>
<ul>
<li><strong>One database for everything.</strong> Postgres stores documents and chunks, a <code>tsvector</code> column with a GIN index handles keyword search, and a <code>vector(768)</code> column with an HNSW index handles semantic search. One service to run and back up, and a document and its chunks update in one transaction.</li>
<li><strong>Chunks that a human would cite.</strong> The chunker makes one chunk per heading section (h1 to h3) instead of a fixed token window. Long sections split at blank lines, never inside a code block, so a chunk never holds half an example.</li>
<li><strong>One anchor rule for Python and the web.</strong> The ingestion code and the Next.js site share the same heading-id rule, so a citation always lands on the right section.</li>
<li><strong>Local first.</strong> Embeddings and answers run on Ollama. No API key is needed to run the project.</li>
</ul>
<h2>Technology stack</h2>
<table>
<thead>
<tr>
<th>Layer</th>
<th>Technology</th>
</tr>
</thead>
<tbody><tr>
<td>Backend API</td>
<td>Python 3.12, FastAPI, streaming answers over Server-Sent Events</td>
</tr>
<tr>
<td>Models</td>
<td>Ollama with <code>nomic-embed-text</code> (embeddings) and <code>qwen3:8b</code> (answers)</td>
</tr>
<tr>
<td>Search</td>
<td>PostgreSQL 17 + pgvector, Postgres full-text search, Reciprocal Rank Fusion</td>
</tr>
<tr>
<td>Reranking</td>
<td><code>cross-encoder/ms-marco-MiniLM-L6-v2</code> via sentence-transformers</td>
</tr>
<tr>
<td>Frontend</td>
<td>Next.js site and chat; the chat engine itself has no React</td>
</tr>
<tr>
<td>Voice (optional)</td>
<td>Pipecat, Silero VAD, Whisper large-v3-turbo, Kokoro-82M</td>
</tr>
<tr>
<td>Observability</td>
<td>Langfuse tracing</td>
</tr>
<tr>
<td>Deployment</td>
<td>Vercel, Modal (API and T4 GPU), Neon, Docker Compose, Helm, Terraform</td>
</tr>
<tr>
<td>Tooling</td>
<td>uv, pytest, ruff, GitHub Actions</td>
</tr>
</tbody></table>
<p>The Claude API is supported for <strong>local development only</strong>. A server started in production mode refuses to enable it, and the public site uses the open <code>qwen3:8b</code> model.</p>
<h2>How the RAG pipeline works</h2>
<p>When you ask a question, the API runs these steps in order:</p>
<ol>
<li><strong>Guard.</strong> A rules-based input guard checks the question and every earlier question in the conversation for injection, jailbreak and prompt-extraction attempts.</li>
<li><strong>Rewrite.</strong> If this is a follow-up like "How do I evaluate it?", the model rewrites it into a standalone question ("How do I evaluate a RAG pipeline?") before searching. The rewritten question goes through the guard again.</li>
<li><strong>Retrieve.</strong> Hybrid search finds candidates, and the cross-encoder reorders them.</li>
<li><strong>Gate.</strong> If even the best match is weakly similar, the API refuses.</li>
<li><strong>Generate.</strong> The model answers from numbered sources and must cite them with <code>[n]</code> markers.</li>
<li><strong>Check.</strong> An output check runs on the answer before the stream ends.</li>
</ol>
<p>The answer streams to the browser as Server-Sent Events: the sources first, then the text, then a final event with details like the best similarity score.</p>
<h2>Hybrid search, reranking and confidence thresholds</h2>
<h3>Hybrid search with RRF</h3>
<p>Keyword search finds exact terms like "RRF" or "pgvector" that embeddings can blur. Vector search finds paraphrases like "make answers cite sources" that keywords miss. Cairnly runs both and merges them with <strong>Reciprocal Rank Fusion</strong>, which uses only the rank of each result. That means the two very different score types never need to be calibrated against each other.</p>
<p>The whole merge is a few lines in <code>search.py</code>:</p>
<pre><code class="language-python">RRF_K = 60  # the constant from the original RRF paper
CANDIDATES = 30  # how deep each list goes before fusion

def rrf(*ranked_lists: list[int], k: int = RRF_K) -&gt; dict[int, float]:
    scores: dict[int, float] = {}
    for ranked in ranked_lists:
        for rank, chunk_id in enumerate(ranked, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0.0) + 1 / (k + rank)
    return scores
</code></pre>
<h3>Cross-encoder reranking</h3>
<p>Hybrid search scores the question and each chunk separately. A cross-encoder reads the question and the chunk <strong>together</strong>, which is more accurate but too slow to run over the whole corpus. So it only reorders the 30 fused candidates.</p>
<p>When this was added, it lifted Recall@5 from 0.88 to 0.94 and MRR from 0.75 to 0.87 on the lessons, for about 17–51 ms per question on a laptop CPU. A larger reranker (<code>bge-reranker-base</code>) was tested too, but it was slower without doing better on this data.</p>
<h3>The confidence gate</h3>
<p>The gate is a single comparison in <code>answer.py</code>:</p>
<pre><code class="language-python">confident = bool(result.hits) and result.top_similarity &gt;= kb.min_score
</code></pre>
<p>The interesting part is how <code>min_score</code> was chosen. It is set per knowledge base in <code>kbs.yaml</code>, and it comes from measurement, not a guess:</p>
<pre><code class="language-yaml">lessons:
  title: AI engineering lessons
  path: content
  min_score: 0.66
  prompt: answer-learner-v2
  rerank: true
  questions: evals/questions
</code></pre>
<p>The eval runs real questions and off-topic questions (piano practice, sourdough, Wi-Fi settings and so on) through the gate at different cutoffs:</p>
<table>
<thead>
<tr>
<th>min_score</th>
<th>Real questions refused</th>
<th>Off-topic questions answered</th>
</tr>
</thead>
<tbody><tr>
<td>0.64</td>
<td>2.1% (13 of 627)</td>
<td>8.0% (12 of 150)</td>
</tr>
<tr>
<td><strong>0.66</strong></td>
<td><strong>5.1% (32 of 627)</strong></td>
<td><strong>2.7% (4 of 150)</strong></td>
</tr>
<tr>
<td>0.68</td>
<td>9.6% (60 of 627)</td>
<td>0%</td>
</tr>
<tr>
<td>0.70</td>
<td>15.3% (96 of 627)</td>
<td>0%</td>
</tr>
</tbody></table>
<p>There is no perfect cutoff. Short questions like "What is a jailbreak?" score close to off-topic ones, so any threshold trades one kind of error for the other. I picked 0.66 because it keeps off-topic answers under a 5% limit while answering 64 more real questions than 0.70. The gate uses the vector similarity, not the reranker score, so the two can be tuned separately.</p>
<h2>Evaluation results and lessons learned</h2>
<p>Here are the current numbers, all from the repository:</p>
<table>
<thead>
<tr>
<th>Metric</th>
<th>Result</th>
</tr>
</thead>
<tbody><tr>
<td>Recall@5 (627 questions: 127 quiz + 500 written)</td>
<td><strong>0.971</strong></td>
</tr>
<tr>
<td>MRR</td>
<td><strong>0.914</strong></td>
</tr>
<tr>
<td>Off-topic questions answered</td>
<td>2.7% (4 of 150)</td>
</tr>
<tr>
<td>Faithfulness (40 sampled questions, from M2)</td>
<td>97.5%</td>
</tr>
<tr>
<td>Follow-up questions (15 cases)</td>
<td>Rewritten follow-ups retrieve as well as standalone questions (0.867 to 0.933 across runs)</td>
</tr>
</tbody></table>
<p>CI runs this eval on every build with real embeddings and <strong>fails the build</strong> if a metric drops below the limits in <code>evals/gates.yaml</code>:</p>
<pre><code class="language-yaml">lessons:
  recall@5: 0.94
  mrr: 0.88
  false_refusal_rate: 0.09
  false_accept_rate: 0.05
</code></pre>
<p>Some limitations I want to be clear about:</p>
<ul>
<li><strong>The faithfulness score is from M2 and has not been re-run since.</strong> The judge was the same local model that wrote the answers (<code>qwen3:8b</code>), and models tend to grade themselves leniently. A stronger, different judge would be a better setup.</li>
<li><strong>The first eval set was too easy.</strong> The original 127 quiz questions were long and precise, so they scored well above anything a learner types in a hurry. Adding 500 written questions (3 to 5 per lesson, including very short ones) showed that the old 0.70 gate was refusing real questions like "What is a jailbreak?". The gate moved down to 0.66 after that measurement.</li>
<li><strong>Ten written questions are still real retrieval misses</strong>, and four off-topic questions still reach the model.</li>
</ul>
<p>The biggest lesson for me: an eval set is only as useful as how closely it matches real questions.</p>
<h2>Voice AI and conversational learning</h2>
<p>Cairnly also has an optional voice mode. You click the microphone, ask a question out loud, and hear the answer with live captions.</p>
<p>It runs as a separate voice server with one Pipecat pipeline per call:</p>
<pre><code class="language-text">caller audio -&gt; Silero VAD -&gt; Whisper large-v3-turbo -&gt; brain -&gt; Kokoro-82M -&gt; caller
</code></pre>
<p>The important design choice is that the "brain" runs no model of its own. Each spoken turn goes to the same <code>/ask</code> endpoint as the text chat, so the guard, the limits, the confidence gate, the citations and the saved conversation all work the same way. Voice does not get a second copy of the logic.</p>
<p>A holding line ("Let me check the lessons.") plays while the answer is looked up, the answer is spoken sentence by sentence as tokens stream, and you can talk over the bot to interrupt it. On synthetic speech in ten accents, Whisper had a 1.5% word error rate, and locally the first words play about 4.6 seconds after you stop talking. Voice mode is self-hosted and needs about 2 GB of speech models.</p>
<h2>Observability, security and deployment</h2>
<h3>Tracing</h3>
<p>Every question can create one Langfuse trace with steps for the guard, rewrite, retrieval, generation and output check. By default, traces hold <strong>no question or answer text</strong>, only sizes, hashes, citations and timings. Full text is only recorded when <code>TRACE_CONTENT=1</code> is set for local debugging.</p>
<h3>Security</h3>
<p>The input guard blocks injection, jailbreak, prompt-extraction and junk input. Since the site teaches prompt injection, asking <strong>about</strong> an attack ("What is a jailbreak?") must still pass. The results are published, including the weak spots:</p>
<table>
<thead>
<tr>
<th>Set</th>
<th>Real questions blocked</th>
<th>Attacks missed</th>
</tr>
</thead>
<tbody><tr>
<td>Tuning set</td>
<td>0 of 955</td>
<td>9 of 155 (5.8%)</td>
</tr>
<tr>
<td>Fresh attacks never used for tuning</td>
<td>0 of 11</td>
<td>8 of 11 (73%)</td>
</tr>
</tbody></table>
<p>So the guard is a cheap first filter, not a wall. Behind it, an output check withdraws any answer that repeats 8 or more words in a row from the system prompt, and flags answers that have no citations. Rate limits (10 questions a minute and 200 a day per IP, at most 4 answers at the same time) keep the free deployment free.</p>
<h3>Deployment</h3>
<p>The public site runs on free and hobby plans: Next.js on Vercel, the API on Modal (one container that switches off when idle), the models on a Modal T4 GPU, and Postgres with pgvector on Neon. The repository also includes Docker Compose, a Helm chart tested on kind, and Terraform for AWS (ECS Fargate, RDS, ALB) as a documented alternative. I moved away from AWS as the default because idle, that setup would cost about $95 a month.</p>
<h2>How to run Cairnly locally</h2>
<p>You need <a href="https://docs.astral.sh/uv/">uv</a>, Docker with Compose v2, Node.js 20+ and <a href="https://ollama.com">Ollama</a>. About 8 GB of free RAM runs <code>qwen3:8b</code>. No API key is needed.</p>
<pre><code class="language-bash">git clone https://github.com/CherukuriSiva/Cairnly.git cairnly &amp;&amp; cd cairnly

# Models (about 5.5 GB in all)
ollama pull nomic-embed-text
ollama pull qwen3:8b

# Settings and database (Postgres 17 + pgvector on port 5434)
cp .env.example .env
docker compose up -d db

# API dependencies, tables and the lesson index
uv sync --extra rerank
uv run cairnly migrate
uv run cairnly ingest --kb lessons

# Site dependencies
cd web &amp;&amp; npm install &amp;&amp; cd ..
</code></pre>
<p>Then run the API and the site in two terminals:</p>
<pre><code class="language-bash">uv run cairnly serve      # API on http://localhost:8010
cd web &amp;&amp; npm run dev     # site and chat on http://localhost:3010
</code></pre>
<p>Open <code>http://localhost:3010/ask</code> and click a starter question. To run the evals yourself:</p>
<pre><code class="language-bash">uv run cairnly eval --kb lessons --gate evals/gates.yaml
</code></pre>
<p>The tests use fake embedders and LLMs, so <code>uv run pytest -m "not db"</code> runs without Ollama or any service.</p>
<h2>Future roadmap</h2>
<p>Milestones M1 to M4 are done: core RAG, evals and the CI gate, the public learning site, and production hardening. These are planned next and <strong>not built yet</strong>:</p>
<table>
<thead>
<tr>
<th>Milestone</th>
<th>Plan</th>
</tr>
</thead>
<tbody><tr>
<td>M5</td>
<td>A study-planner agent with tools, and an MCP server for Claude and Cursor</td>
</tr>
<tr>
<td>M6</td>
<td>MLOps: pipelines, MLflow, a LoRA fine-tune experiment, drift monitoring</td>
</tr>
<tr>
<td>M7</td>
<td>Platform: auth, multi-tenancy, RBAC, audit log, PII redaction, response cache</td>
</tr>
<tr>
<td>M8</td>
<td>Reliability: Prometheus and Grafana, SLOs and alerts, load tests, runbooks</td>
</tr>
</tbody></table>
<h2>How to contribute</h2>
<p>There are two ways to help:</p>
<ul>
<li><strong>Lessons:</strong> fix or improve content in <code>content/</code>. Write in your own words, one lesson per pull request, and run <code>cd web &amp;&amp; npm test</code> to check links, anchors and quizzes.</li>
<li><strong>Code:</strong> add a test for every behavior you change. Retrieval or prompt changes must keep the eval gate green, and a significant design choice gets a short ADR in <code>docs/adr/</code>.</li>
</ul>
<p>Issues labelled <code>good first issue</code> are a good place to start. For anything bigger than a typo, open an issue first. For security problems, please use GitHub's private "Report a vulnerability" option instead of a public issue.</p>
<h2>Conclusion</h2>
<p>Cairnly is my attempt to put learning and building in the same place. You can read the lessons and ask the tutor, or you can open the code and see how a RAG app handles retrieval, refusals, evals, safety, tracing and deployment, with the real numbers and the known gaps written down.</p>
<p>If you are building your own RAG system, the <a href="https://github.com/CherukuriSiva/Cairnly/tree/main/docs/adr">design decisions (ADRs)</a> and the plain-English write-ups in <code>docs/</code> explain why each piece works the way it does, even if you never run the project.</p>
<p>If you try it, I'd love your feedback:</p>
<ul>
<li>Star or fork the <a href="https://github.com/CherukuriSiva/Cairnly">GitHub repository</a></li>
<li>Ask the <a href="https://cairnly-blond.vercel.app/ask">tutor</a> a question and tell me where it gets things wrong</li>
<li>Open an issue or pick a <code>good first issue</code></li>
</ul>
<p>Thanks for reading.</p>
]]></content:encoded></item></channel></rss>