Anyone can build an AI demo in an afternoon. Shipping AI features to production reliably is a completely different exercise. Here's the checklist we run on every AI project.
Before you build
1. Is this actually an AI problem? Not every problem needs AI. Classification, extraction, summarisation, semantic search, drafting — yes. Exact-accuracy math, guaranteed-format output, sub-100ms latency — usually not. Ask honestly.
2. What does "good enough" look like? Define success in measurable terms before writing prompts. "Users like it" is not a target. "80%+ of extracted invoice fields correct" is.
3. Who owns quality post-launch? AI features degrade if nobody watches them. Assign an owner before you ship.
While you build
4. Evaluation harness first. Before you tune prompts, build a test set of 50–200 input-output pairs. Every prompt change gets scored against it. If you don't measure, you're vibing.
5. Prompt versioning. Prompts change frequently. They need to be versioned, tested, and rolled back like any code. Store them separately from application code so non-engineers can iterate.
6. Fallback logic. Every LLM call needs: retry on rate limit, retry on timeout, fallback to cheaper model, fallback to non-AI path when everything fails. Never trust a single API call to succeed.
7. Structured outputs. When you need JSON or specific formats, use the provider's structured output mode. Free-form prompt engineering for format compliance loses to native structured output every time.
8. Cost caps. Set per-user and per-day cost limits. Alert if usage spikes. AI costs go out of hand only when nobody's watching — make sure someone is.
Safety
9. Input validation. Users will send prompts that try to jailbreak, exfiltrate data, or waste tokens. Validate inputs, rate-limit, and log suspicious patterns.
10. Output filtering. Screen model outputs for sensitive data leakage (PII, credentials, internal info). Filter or redact before showing to users.
11. Human-in-the-loop where stakes matter. For high-stakes decisions (legal, medical, financial recommendations), require a human review step. Never let an LLM make the final call on money or health.
12. Prompt injection defence. User inputs get placed inside your system prompts. Malicious users will try to override your instructions. Use structured prompts, sanitise user inputs, and set clear boundaries in system messages.
Monitoring
13. Log everything. Full request/response logs for every LLM call. Retention long enough to debug week-old issues. Search-friendly enough to find patterns.
14. Track quality metrics. Sample outputs, run them through your eval harness, alert if quality drops. Model providers occasionally deprecate versions or change behaviour — you find out when quality drops silently.
15. Track cost per user. Per-user cost tracking prevents runaway spending. A single power user can rack up $500/day if unchecked.
16. User feedback loop. Thumbs up/down on every AI response. Ship it. Read it. Adjust prompts. This is the fastest quality improvement mechanism.
Deployment
17. Feature flag it. Ship AI features behind a flag. Enable for 5% of users first. Watch metrics. Roll out gradually.
18. Model version pinning. Pin the exact model version (gpt-4o-2024-08-06, not just gpt-4o). Providers deprecate models. Your feature can break silently on the deprecation date if you're using auto-updating aliases.
19. Rate limit handling. Provider rate limits WILL be exceeded during traffic spikes. Queue, retry with backoff, degrade gracefully.
20. Fallback UX. When the AI is slow (>3s response) or fails, what does the user see? Design this before it becomes an emergency.
Ongoing
21. Retrain evaluation set as reality shifts. Your test set from 3 months ago may not reflect current user behaviour. Update quarterly.
22. Watch for model provider changes. Providers release new models constantly. Test whether upgrading helps. Sometimes newer models are cheaper AND better.
23. Reprompt engineering as costs shift. Prompt caching, structured outputs, and reasoning models change the cost/quality tradeoff. Revisit every 6 months.
What separates production from demo
- Demos hit the happy path. Production handles the 20% edge cases.
- Demos use whatever prompt worked once. Production evaluates every prompt change.
- Demos cost pennies during development. Production costs need to be modelled at real scale.
- Demos can fail silently and nobody notices. Production failures have downstream impact.
If you're evaluating an AI feature build, ask the team building it whether they've addressed the items above. The gap between "we can build this" and "this will work in production" is exactly the items on this checklist.
We ship production AI features for clients. Book a consultation and we'll walk through your specific use case.