Shipping an AI App

The demo went beautifully. Your assistant answered every question your manager typed, and by Friday it is “live” for the whole department. On Monday someone pastes a customer’s IC number into it. On Tuesday someone asks it to write a wedding speech, and it does — on the company account. On Wednesday finance asks why the API bill tripled, and you discover you have no record of who asked what, how long it took, or what it cost. None of these is a model problem. Every one is a shipping problem, and every one is cheap to prevent if you build the boring parts first.

From script to app

On earlier pages you called a model from a script (Calling LLM APIs), grounded it in documents (Retrieval-Augmented Generation) and let it use tools (Tool Use and Agents). A script trusts its only user: you. An app has to assume users who make mistakes, users who test its limits, and a world where the network fails and the provider is sometimes busy. The core model call stays the same; what changes is everything around it. Tap each stage of a single request’s journey.

We will build each stage as a small Python file — settings.py, guardrails.py, monitor.py — and then join them in a tiny web app. Each piece is tested on its own, which is the main trick of shipping: small parts you have seen working.

Configuration and secrets

Anything that differs between your laptop and the server — the API key, the model name, limits — belongs in environment variables, not in code. This is an old rule of web development (the “twelve-factor app” calls it storing config in the environment), and it matters doubly for AI apps because the key is tied to billing. During development, keep the values in a .env file and make sure Git never sees it:

# .env  -- real values, on your machine only
ANTHROPIC_API_KEY=paste-your-key-here
APP_MODEL=claude-opus-5-5
APP_MAX_INPUT_CHARS=2000

# .gitignore  -- list it BEFORE your first commit
.env
.venv/
calls.log
__pycache__/

Commit a copy called .env.example with the names but blank values, so a colleague knows what to set. Then read the settings in one place:

# settings.py - every setting comes from the environment, never from the code
import os
from dotenv import load_dotenv

load_dotenv()                                   # copies .env into os.environ (if the file exists)
MODEL = os.environ.get("APP_MODEL", "claude-opus-5-5")   # check the provider's docs for current names
MAX_INPUT_CHARS = int(os.environ.get("APP_MAX_INPUT_CHARS", "2000"))
HAS_KEY = bool(os.environ.get("ANTHROPIC_API_KEY"))

if __name__ == "__main__":
    print("model:", MODEL, "| input limit:", MAX_INPUT_CHARS, "| key present:", HAS_KEY)

model: claude-opus-5-5 | input limit: 2000 | key present: True

We ran this in a terminal with no key set, so “key present: True” proves load_dotenv() found the .env file. By default it does not overwrite variables that are already set, so on a server you can skip the file entirely and set real environment variables through your hosting platform’s settings or secrets manager. Notice the script prints whether a key exists, never the key itself. If a key ever lands in a commit, deleting the file is not enough — it lives on in Git history. Revoke it at the provider and make a new one.

Guardrails

Guardrails are the checks around the model that do not depend on the model behaving. Some are a few lines of code; some are a person. A sensible first set:

# guardrails.py - cheap checks before and after the model call
import re

MYKAD = re.compile(r"\b\d{6}-?\d{2}-?\d{4}\b")        # Malaysian IC number pattern
PHONE = re.compile(r"\b01\d-?\d{7,8}\b")              # Malaysian mobile number pattern

def check_input(text, max_chars=2000):
    text = text.strip()
    if not text:
        return False, "Please type a question."
    if len(text) > max_chars:
        return False, f"Please keep questions under {max_chars} characters."
    if MYKAD.search(text) or PHONE.search(text):
        return False, "Please remove IC or phone numbers before asking."
    return True, text

def check_output(answer):
    if not answer.strip():
        return "Sorry, I could not produce an answer. A colleague will follow up."
    return PHONE.sub("[number removed]", MYKAD.sub("[IC removed]", answer))

if __name__ == "__main__":
    for q in ["", "x" * 2500, "My IC is 900101-14-5678, am I covered?",
              "  What are the opening hours on Sunday?  "]:
        print(check_input(q)[0], "|", check_input(q)[1][:60])
    print(check_output("Call Aminah at 012-3456789 for help."))

False | Please type a question.
False | Please keep questions under 2000 characters.
False | Please remove IC or phone numbers before asking.
True | What are the opening hours on Sunday?
Call Aminah at [number removed] for help.

What each check buys you:

• Input limits. A length cap protects your bill and your context window from someone pasting a whole book. Rejecting inputs that look like IC or phone numbers means personal data never leaves your server. (These two regular expressions are deliberately simple; they will miss some formats and occasionally catch innocent numbers. Tune them on real, anonymised examples.)
• Output checks. Never show raw model output without a look. Here we mask number patterns and replace empty answers with an honest fallback. For a RAG app, a stronger check is that every answer cites at least one retrieved source.
• Scope and refusals. The system prompt should say what the assistant is for and what to do outside it: “If the question is not about our opening hours, delivery or returns, say you can only help with those.” A polite, consistent refusal is a feature, not a failure.
• Human review. Decide in advance which answers go straight to users and which a person checks first. Anything with legal, medical, financial or employment consequences belongs in the second group, as does anything an agent would do rather than say.

Consensus No single guardrail is reliable on its own — pattern filters miss things, models can be talked past their instructions, and reviewers get tired. Layers that fail differently are what make the whole dependable.

Logging and monitoring

Once real people use your app, you need to know how it is doing without asking them. Monitoring starts with one log line per model call, recording numbers: did it succeed, how long did it take, how many tokens went in and out, and roughly what did it cost.

# monitor.py - one log line per model call: numbers, never the user's words
import json, logging, time, uuid

log = logging.getLogger("calls")                 # our own logger, separate from libraries'
log.setLevel(logging.INFO)
log.addHandler(logging.FileHandler("calls.log"))
PRICE_IN, PRICE_OUT = 5.00, 25.00   # USD per million tokens - EXAMPLE figures; use your provider's price page

def log_call(started, status, input_tokens=0, output_tokens=0):
    cost = (input_tokens * PRICE_IN + output_tokens * PRICE_OUT) / 1_000_000
    record = {"id": uuid.uuid4().hex[:8], "status": status,
              "latency_s": round(time.perf_counter() - started, 2),
              "in_tok": input_tokens, "out_tok": output_tokens, "cost_usd": round(cost, 5)}
    log.info(json.dumps(record))
    return record

if __name__ == "__main__":
    t0 = time.perf_counter()
    print(log_call(t0, "ok", input_tokens=1850, output_tokens=240))
    print(log_call(t0, "rate_limited"))

{'id': '833d8972', 'status': 'ok', 'latency_s': 0.0, 'in_tok': 1850, 'out_tok': 240, 'cost_usd': 0.01525}
{'id': '725f8376', 'status': 'rate_limited', 'latency_s': 0.0, 'in_tok': 0, 'out_tok': 0, 'cost_usd': 0.0}

The ids are random, so yours will differ. Three design choices are worth copying:

• No personal data. The record holds no question, no answer, no name. If you need examples for debugging, collect them separately, with consent, for a limited time, and anonymise them.
• A named logger. Our first version used Python’s catch-all root logger, and the log file promptly filled with the HTTP library’s own messages as well as ours. A logger called "calls" keeps your file to your records.
• Prices as labelled constants. The prices above are example figures for the arithmetic only. Real prices differ by model and change over time, so read them from your provider’s pricing page and keep them in one place.

From these lines you can answer the questions that matter in week one: how many calls failed and why, how slow the slowest ten per cent were, and what yesterday cost. Watch the trend, set an alert for error spikes, and set a hard monthly spending limit on the provider account as a backstop.

Privacy and Malaysia’s PDPA

If your app handles information about identifiable people in a commercial setting, Malaysia’s Personal Data Protection Act 2010 applies to you, and its 2024 amendments added duties such as notifying data breaches and, for many organisations, appointing a data protection officer As of 2026. In practice for an AI app: collect the minimum, tell users what happens to what they type (including that it is sent to an external AI provider, and where that provider processes it), keep logs free of personal data, delete what you no longer need, and check your organisation’s approved-services list before connecting to any provider. The Practitioner page has a plain-language summary of Malaysia’s PDPA in brief; for a real product, talk to whoever handles data protection in your organisation. This is general information, not legal advice.

A minimal web interface

Now join the pieces. Flask is a small Python web framework; here it exposes one address, /ask, that takes a question as JSON and returns an answer. Flask is not in this track’s standard set of packages, so install it first with pip install flask.

# app.py - a tiny web API around the model, with guardrails and logging
import time
import anthropic
from flask import Flask, request, jsonify
from settings import MODEL, MAX_INPUT_CHARS
from guardrails import check_input, check_output
from monitor import log_call

app = Flask(__name__)
client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY
SYSTEM = "You answer questions about our shop's opening hours, delivery and returns. If unsure, say so."

def ask_model(question):
    response = client.messages.create(model=MODEL, max_tokens=2000, system=SYSTEM,
                                      messages=[{"role": "user", "content": question}])
    text = "".join(b.text for b in response.content if b.type == "text")
    if response.stop_reason == "max_tokens":
        text += " [answer cut short]"
    return text, response.usage.input_tokens, response.usage.output_tokens

@app.post("/ask")
def ask():
    data = request.get_json(silent=True) or {}
    ok, cleaned = check_input(str(data.get("question", "")), MAX_INPUT_CHARS)
    if not ok:
        return jsonify(error=cleaned), 400
    started = time.perf_counter()
    try:
        text, tin, tout = ask_model(cleaned)
    except anthropic.RateLimitError:
        log_call(started, "rate_limited")
        return jsonify(error="Busy - please try again in a minute."), 503
    except anthropic.APIStatusError as e:
        log_call(started, f"api_error_{e.status_code}")
        return jsonify(error="Something went wrong on our side."), 502
    log_call(started, "ok", tin, tout)
    return jsonify(answer=check_output(text))

# ui.py - a one-box chat page in Streamlit (not executed for this page)
import streamlit as st
from app import ask_model
from guardrails import check_input, check_output

st.title("Kedai helper")
question = st.chat_input("Ask about opening hours, delivery or returns")
if question:
    st.chat_message("user").write(question)
    ok, cleaned = check_input(question)
    if not ok:
        st.chat_message("assistant").write(cleaned)
    else:
        text, _, _ = ask_model(cleaned)
        st.chat_message("assistant").write(check_output(text))

The order inside ask() is the chain from the top of the page: check input, call the model, handle failure, log, check output. Errors from the provider become friendly messages with honest HTTP status codes (503 “busy”, 502 “our side”), and the user never sees a Python traceback. max_tokens=2000 suits short shop answers; if a reply is ever cut short, the app says so instead of pretending it is complete.

You do not need to start a server to test this. Flask’s test client sends pretend requests straight to the function, and here we also swap ask_model for a stand-in so no API call is made:

# test_app.py - exercise the route with Flask's test client: no server, no real API call
import app

app.ask_model = lambda q: ("We open 9am-6pm on Sundays. Questions? Call 012-3456789.", 412, 18)
web = app.app.test_client()
for body in [{"question": "What time do you open on Sunday?"},
             {"question": "My IC is 900101-14-5678"},
             {}]:
    r = web.post("/ask", json=body)
    print(r.status_code, r.get_json())

200 {'answer': 'We open 9am-6pm on Sundays. Questions? Call [number removed].'}
400 {'error': 'Please remove IC or phone numbers before asking.'}
400 {'error': 'Please type a question.'}

We also posted a question through the real ask_model with a placeholder key. The provider refused it, the route returned 502 {'error': 'Something went wrong on our side.'}, and calls.log gained this line — the failure is recorded, the user is not shown the details:

{"id": "9fe37af4", "status": "api_error_401", "latency_s": 0.38, "in_tok": 0, "out_tok": 0, "cost_usd": 0.0}

To try it for real on your own machine, run flask --app app run and send a request with a tool such as curl to http://127.0.0.1:5000/ask. That built-in server is for development only. To put the app in front of other people you would run it under a production web server (such as Gunicorn on Linux or Waitress on Windows) behind HTTPS, with sign-in so only your intended users can reach it and a per-user limit on requests, on whatever hosting your organisation already uses. Those steps are general web practice rather than anything AI-specific, and your IT team will have standards for them.

Evaluate before release, and control cost

On Measuring Models you learned never to trust a model you have not tested on held-out examples. The same rule holds for an assistant. Before launch, and before every change of prompt, model or document set, run a fixed list of realistic questions with known good answers — an evaluation set — and compare scores. Include awkward cases on purpose: questions the documents do not answer (the right reply is “I don’t know”), off-topic requests, inputs with personal data, and a few written in Bahasa Melayu or mixed language if your users write that way. The Builder Capstone includes a small scoring script you can adapt.

Cost is mostly tokens, so the levers are about tokens:

• Cap the output. Set max_tokens to suit the job and handle stop_reason == "max_tokens" as we did. Long answers cost more and are read less.
• Send less input. Retrieve the three most relevant passages, not the whole handbook. Trim old conversation turns. Use client.messages.count_tokens(...) to measure before you guess.
• Cache the repeated part. If every request starts with the same long instructions or FAQ, prompt caching lets the provider reuse it. Cached reads are billed at a small fraction of the normal input price, and the default cache lasts about five minutes after last use As of 2026.
• Use a smaller model where quality holds. Providers offer faster, cheaper tiers. Run your evaluation set on each; if the smaller model scores the same on your questions, use it, and keep the larger one for the hard cases.

# cached.py - reuse a long, unchanging prefix (e.g. your FAQ) across requests
import anthropic

MODEL = "claude-opus-5-5"   # check the provider's docs for current model names
client = anthropic.Anthropic()
FAQ = open("faq.md", encoding="utf-8").read()       # the same long text on every request

response = client.messages.create(
    model=MODEL, max_tokens=2000,
    cache_control={"type": "ephemeral"},             # ask the API to cache the repeated prefix
    system="Answer only from this FAQ. If it is not covered, say so.\n\n" + FAQ,
    messages=[{"role": "user", "content": "Do you deliver on public holidays?"}],
)
u = response.usage
print("input:", u.input_tokens, "cache write:", u.cache_creation_input_tokens,
      "cache read:", u.cache_read_input_tokens, "output:", u.output_tokens)

With a placeholder key this fails with AuthenticationError: Error code: 401, confirming the call shape. With a real key, the first request reports tokens written to the cache and later ones within the cache lifetime report tokens read from it. Prefixes shorter than a model-specific minimum (several hundred tokens or more) are simply not cached, without an error — so if both cache numbers stay at zero, your prefix is too short to benefit.

A pre-launch checklist

Work down this list before anyone outside your team uses the app. Each step is quick; skipping one is how the Monday-to-Wednesday story at the top of this page happens.

1. Purpose and scope written downOne paragraph: who it is for, what it answers, what it must refuse, and who owns it. The system prompt and the interface both reflect it.
2. Approval and data checkThe AI provider is on your organisation’s approved list; you know what data users will type and where it is processed; PDPA duties are understood.
3. Secrets out of codeKeys and settings in environment variables; .env in .gitignore; .env.example committed; spending limit set on the provider account.
4. Guardrails in placeInput length and personal-data checks, output checks, a polite refusal for off-topic requests, friendly error messages, and a turn limit for any tool loop.
5. Evaluation passedYour evaluation set, including awkward and “I don’t know” cases, scores at or above the bar you agreed in advance — and the results are saved for comparison next time.
6. Logging and alerts onOne log line per call with status, latency, tokens and cost; no personal data; someone is told when errors or spending spike.
7. Humans in the right placesHigh-stakes answers and every action with side effects are reviewed; users can see it is AI, know it can be wrong, and can reach a person.
8. Small, reversible launchStart with a pilot group, read the logs daily for the first week, and keep a way to switch the assistant off quickly.

Key terms on this page

Sources & further reading

• Anthropic, Prompt caching (Claude API documentation, 2026).
• Pallets, Flask Quickstart (Flask documentation).
• Python Software Foundation, Logging HOWTO (Python documentation).
• python-dotenv (project documentation).
• Adam Wiggins, The Twelve-Factor App: Config (2011).
• OWASP GenAI Security Project, OWASP Top 10 for LLM Applications (2025).
• Personal Data Protection Department Malaysia, Official website — the Act, amendments and guidelines.