← Ben Knox-Johnston

Production RAG, shown working.

Retrieval Augmented Generation: AI answers grounded in real data, with receipts. I built this end to end in a 2 day technical assessment, and every claim on this page links to the code, the decision, or the measurement behind it.

from empty repo to submission
2 days
from empty repo to submission
on the grounding eval, refusals included
18/18
on the grounding eval, refusals included
automated tests
181
automated tests
reviewed pull requests
22
reviewed pull requests
Read the codeRun it in 3 commands
An answer with a citation open, showing the exact transcript excerpt

One meeting's journey through the pipeline

This is what happens to a meeting inside the system. Every artefact below is real data pulled from the running database.

  1. 01

    A meeting arrives.

    Upload a Teams transcript, or paste one in. This journey follows a real line from the demo pack:

    Aisha Balogun: "Some numbers to make it concrete. We've pushed a little over twelve thousand test messages through the staging soak since the end of June — delivery rate 99.7 percent inside thirty seconds..."
    Transcript line, Sprint Retro & Roadmap, 8:16
  2. 02

    It gets split into chunks.

    Along speaker turns, about 1000 characters each, so a chunk holds a piece of conversation rather than a random slice. Aisha's line lands in chunk 11 of Sprint Retro & Roadmap: 807 characters, three speakers, 8:16 to 9:13. One thing to notice: the vendor's name appears nowhere in those 807 characters.

  3. 03

    Each chunk gets its context back.

    At ingest, the model reads the whole transcript and writes one sentence saying where this fragment sits. The sentence it actually wrote for chunk 11:

    "This fragment is part of the Sprint Retro & Roadmap meeting (2026-07-22) where Priya Raman, Aisha Balogun, and Dan Whitmore discuss the successful validation of Corvane Health as the SMS vendor choice..."
    Stored context sentence, chunk 11. There's the vendor's name the chunk itself never says.
  4. 04

    It's indexed twice.

    A keyword index and a 1024 dimension embedding, both built over the context sentence plus the spoken text. Findable by meaning and by name.

  5. 05

    A question comes in.

    "Was choosing Corvane the right call in hindsight?" Both searches run at the same time, each brings back its best 30 chunks, and the lists merge by rank. Chunk 11 surfaces because its context sentence knows who the question is about.

  6. 06

    The model answers from sources only.

    The top 20 chunks go in, each labelled with meeting, time and speakers. The rules say cite every claim, and if the facts aren't there, say so.

  7. 07

    The answer shows its receipts.

    Click a citation and read the actual moment in the meeting. That's the whole promise.

    Citation chips under an answer, one open on its transcript excerpt

Step 3 exists because of a failure. The next section is that story.

The AI cited a number that wasn't in the meeting

Before

delivery rate 99.8% cited, wrong

After

delivery rate 99.7% cited, correct

During testing, I asked about the vendor choice and the answer came back with a delivery rate of 99.8 percent, citation attached. The transcript says 99.7. A wrong number with a receipt next to it is the worst failure this kind of system can have, and my eval is what caught it.

The diagnosis took real digging. Chunk 11 holds the real figure, but as the journey above showed, it never names the vendor. So for a question about the vendor, neither keyword search nor vector search could surface it. The model had the question, no delivery rate anywhere in its sources, and it filled the gap with a plausible number.

The fix is contextual retrieval, Anthropic's published approach for exactly this: at ingest, every chunk gets one sentence of context written with the whole transcript in view. That sentence names the vendor, both indexes include it, and chunk 11 now surfaces for the question it always should have answered. The right figure comes back every time.

The part I'd want you to notice: I could have widened the eval's accepted answers until the test went green, and I actually started to, then stopped, because that is tuning the test to the answer. The test stayed exactly as it was. The pipeline changed instead, and the score returned to 18 of 18 on repeated runs.

“Seventeen or eighteen out of eighteen with the reason written down is worth more than eighteen every time.”
From the decision record, written that day →

Trust by design

When the answer isn't in the meetings, it says so. That behaviour is deliberate, and it's tested.

It refuses rather than invents.

Ask about something that was never discussed and it tells you it couldn't find it, points at the closest real thing it did find, and attaches no citation. Three of the eval's eighteen questions exist purely to keep that true.

Nothing from a user is trusted.

Transcript text comes from whoever uploaded the file. Everything is escaped before it reaches a prompt, and every kind of line break is stripped out of speaker names at parse, because a line break inside a name would read like somebody else talking. That defence came out of a security review, and then it became a test.

Found by audit, fixed the same day.

An independent audit caught the model occasionally writing a citation as a raw source tag instead of a numbered chip. The fix went in at three layers: a tighter prompt rule, a repair step, and a filter that catches tokens split across the stream. The screenshots on this page were re-taken with an automated check that no raw token can ship again.

The full list of guardrails, including the limits accepted on purpose, is written down in the repo.

The app declining to answer a question that was never discussed

Measured, honestly

Eighteen questions with known right answers run against the live pipeline: direct facts, threads that cross meetings, date questions, a follow up that needs chat memory, and three questions with no answer in the data, where refusing is the pass.

Scoring is deterministic string matching, and that's legitimate here for one reason: I wrote the demo data, so the ground truth is exact. A judge model would cost more and be less predictable at this scale.

Current score: 18 of 18 on retrieval and 18 of 18 on grounding, re-run in full after every change to the answer path, most recently the night of submission.

And the limits, stated in the decision record rather than discovered by a reviewer: eighteen questions is a small set. The retrieval check confirms the right meeting came back, and the next investment is confirming the exact fact came back too. The eval runs on demand today; in production it gates deploys from CI. Knowing what your eval can't see is part of having one.

$ npm run eval
PASS  vendor-price         fact   retrieval=hit  grounded=yes
PASS  vendor-hindsight     cross  retrieval=hit  grounded=yes
PASS  golive-change        cross  retrieval=hit  grounded=yes
PASS  followup-owner       followup  retrieval=hit  grounded=yes
PASS  unanswerable-hiring  unanswerable  retrieval=hit  grounded=yes
  ...13 more
retrieval hit rate: 18/18
grounded answers:   18/18

Run like production

Everything above is checkable in about a minute each: the CI runs, the PR reviews, the log shape, the script.

Run it yourself

Three commands, and the app opens with nine demo meetings already loaded. API keys are only needed when you start asking questions, and the getting started guide walks the rest.

$ git clone https://github.com/BenKnoxJ/meeting_intelligence.git
$ cd meeting_intelligence
$ ./start.sh
Meeting Intelligence is running: http://localhost:3000

Every decision was written down the day it was made: 23 records, from the stack to the guardrails.

The repository →The decision records →Getting started →Demo guide →