Builder Capstone

Put the whole track together: build and ship a small assistant that answers questions from your own documents, shows where each answer came from, and proves its quality on a test set before anyone else uses it.

Every organisation has a folder of documents that people ask the same questions about: the staff handbook, the club constitution, the product manuals, the course notes. Someone always ends up being the human search engine. Your capstone is to build the assistant that takes over that job — one that answers only from those documents, says which one it used, admits when the answer is not there, and has a score you can show to a sceptical colleague. You already have every piece. This page is the brief.

The brief

Build a question-answering assistant over a small collection of documents you choose — between five and fifty pages of text is plenty. It must:

Choose your documents carefully

Use documents you have the right to use and that contain no personal or confidential data: public material (a government guideline, a public-domain book, your own published writing) or your organisation’s documents only with permission and within its rules on approved AI services. Your API key stays in an environment variable from the first line of code.

Milestones

Work through these in order. Each ends with something that runs, so you are never more than a step away from a working version. Allow anything from a long weekend to a few weeks of evenings.

  1. Write the one-paragraph specWho the assistant is for, which documents, five example questions, what it must refuse. Set up a fresh project folder, virtual environment, .env and .gitignore.
  2. Write the evaluation set firstAbout ten questions with the phrase a correct answer must contain and the document it should cite. Include two or more “not in the documents” questions and one off-topic request. Writing these before the code keeps you honest.
  3. Load and chunk the documentsRead the files, split them into passages of a few paragraphs, and keep each passage’s file name and heading with it. Print a few chunks and read them: bad chunks cause bad answers.
  4. Build retrieval and test it aloneFor each evaluation question, does the right passage appear in the top three results? Fix retrieval before involving a language model at all.
  5. Add the model with citationsSend the retrieved passages with the question; instruct the model to answer only from them, cite file names, and say plainly when the answer is missing. Return the cited files alongside the text.
  6. Turn retrieval into a toolExpose search as a tool in a manual loop with a turn limit and error results, so the model can search again. Log each tool call.
  7. Score it, improve it, score againRun the evaluation script. Change one thing at a time — chunk size, number of passages, prompt wording, model — and keep a table of scores.
  8. Wrap it and ship it to one personAdd the interface, guardrails and call log. Give it to one real user, watch them use it without helping, and note every surprise.

Scoring your assistant

An assistant is only as trustworthy as the test it has passed. Here is a minimal scoring script. The answer function is a stand-in that returns canned replies so you can see the scoring work; replace it with your real assistant, which should return the answer text and the list of files it cited. The handbook and its contents are made up, and there are only three cases to keep it short — yours should have about ten.

# eval_harness.py - score your assistant on a fixed set of questions
EVAL_SET = [   # question, a phrase the answer must contain, the source it should cite
    {"q": "How many days of annual leave do new staff get?", "must": "14", "cite": "hr-handbook.md"},
    {"q": "Who approves overtime claims?", "must": "head of department", "cite": "hr-handbook.md"},
    {"q": "What is the office wifi password?", "must": "don't know", "cite": None},  # not in the docs
]

def answer(question):   # stand-in for your RAG assistant: returns (text, [cited files])
    canned = {
        "How many days of annual leave do new staff get?": ("New staff get 14 days a year.", ["hr-handbook.md"]),
        "Who approves overtime claims?": ("Your manager approves them.", ["hr-handbook.md"]),
        "What is the office wifi password?": ("I don't know - it isn't in the documents.", []),
    }
    return canned[question]

passed = 0
for case in EVAL_SET:
    text, cited = answer(case["q"])
    fact_ok = case["must"].lower() in text.lower()
    cite_ok = (case["cite"] in cited) if case["cite"] else not cited
    passed += fact_ok and cite_ok
    print(f"{'PASS' if fact_ok and cite_ok else 'FAIL'}  fact={fact_ok!s:5} cite={cite_ok!s:5}  {case['q']}")
print(f"\n{passed}/{len(EVAL_SET)} passed")
PASS  fact=True  cite=True   How many days of annual leave do new staff get?
FAIL  fact=False cite=True   Who approves overtime claims?
PASS  fact=True  cite=True   What is the office wifi password?

2/3 passed

Each case checks two things separately: did the answer contain the key fact, and did it cite the right document (or, for the “not in the documents” case, cite nothing)? Keeping them apart tells you what to fix: a citation that is right with a fact that is wrong usually means the model misread a good passage; a wrong citation usually means retrieval failed. The second answer fails because it cites the right file but says “manager” instead of “head of department” — exactly the kind of confident near-miss users would not notice.

Phrase matching is crude: “fourteen days” would fail a test for “14”. That is acceptable for ten questions you wrote yourself — read every failure and decide whether it is a real error or a test that is too strict. Larger projects add human grading or a second model as a judge, each with its own pitfalls; the Measuring Models page explains why a single number can hide as much as it shows.

Rubric

Use this to judge your own project, or swap with a friend and judge each other’s. Each criterion is worth up to 3 points: 1 = attempted, 2 = works, 3 = works and you can show the evidence.

1Answers groundedCorrectness on the evaluation set

3 points: at least eight of ten evaluation questions pass, and you have read every failure and can explain it. Answers come from the documents, not the model’s general knowledge.

2Citations you can checkSources shown and correct

3 points: every answer names its document (and ideally section), and when you open that document the claim is really there. Missing answers are admitted, not invented.

3Tool loop done properlyLimits, errors and logs

3 points: the loop has a turn limit, returns errors as error results, validates tool inputs, and logs each tool call.

4Safe to hand overSecrets, guardrails, privacy

3 points: no key in code or history; input and output checks; logs contain no personal data; the interface says what the assistant is for and that it can be wrong.

5Evidence of iterationYou measured, changed, measured

3 points: a short table showing at least three versions (for example, different chunk sizes or prompts) with their scores, and a sentence on what you learned from each.

6A real user tried itFeedback acted on

3 points: one person outside the project used it unaided; you wrote down what confused them and changed at least one thing as a result.

Fifteen or more out of eighteen is a project you could show in a job interview. Below ten, look first at criteria 1 and 4: an assistant that is wrong or unsafe is not finished, however good it looks.

Stretch goals

Track quiz

Twelve questions across the whole Builder track. If you miss one, the explanation points to the page to revisit.

Builder track check

Why make a separate virtual environment for each project?

Isolation is the point. See The Builder’s Road.

In pandas, what is a DataFrame?

It is the spreadsheet-like table at the heart of pandas. See Working with Data.

Why do you keep a test set aside and not train on it?

Scoring on training data rewards memorising. See The Machine Learning Workflow.

A model scores 99% on training data and 70% on new data. What is the likeliest problem?

A big gap between training and test performance is the signature of overfitting. See Measuring Models.

In a fraud dataset where 1% of cases are fraud, a model that always says “not fraud” is 99% accurate. Which measure exposes it?

It catches none of the fraud, so its recall for fraud is zero. See Measuring Models.

Why must a chat app resend the whole conversation on every API call?

Each request stands alone, which is also why long chats cost more. See Calling LLM APIs.

The model’s reply ends with stop_reason “max_tokens”. What happened?

Raise the limit or ask for a shorter answer, and tell the user if it was cut. See Calling LLM APIs.

What makes two embeddings “close”?

Embeddings place similar meanings near each other, even with different words. See Embeddings and Semantic Search.

A RAG assistant gives a wrong answer with a correct-looking citation. Where should you look first?

Check retrieval and the passage before blaming generation. See Retrieval-Augmented Generation.

In tool use, who runs the tool’s code?

The model only proposes a call; your code decides and acts. See Tool Use and Agents.

An agent reads web pages and can send emails. What is the main defence against prompt injection?

No prompt reliably stops injection; limiting the damage does. See Tool Use and Agents.

What belongs in your app’s call log?

Numbers, not personal data — and never the key. See Shipping an AI App.

// Pause and reflect

  1. Which part of the capstone do you expect to take longest — chunking, retrieval, prompting, evaluation or the interface — and why?
  2. What is one question your assistant should refuse or answer “I don’t know” to, and how will you test that it does?
  3. Looking back over the Builder track, what can you now explain to a colleague that you could not explain before?

Your notes are saved privately in this browser.

Where next?

You can now build, measure and ship a small AI application, which already puts you ahead of many people who talk about AI for a living. Two directions open from here. If you want to understand what happens inside the models you have been calling — neural networks, attention, training and fine-tuning — continue to The Advanced Road. If you would rather keep building, pick a course or book at your level from Courses, Books and Resources, and make your next project slightly harder than this one.

Key terms on this page

Sources & further reading