Designing AI Features That Don't Break in Production
Prompt versioning, structured outputs, cost control, graceful degradation and evaluation — the engineering around a model call that decides whether the feature survives.
Quick answer
Treat a model call like any unreliable third-party dependency: force structured output with a schema and validate it, version prompts as code, set timeouts with a deterministic fallback, cap spend per user and per day, cache aggressively, and run an evaluation set in CI. The model is the easy part; the surrounding engineering decides whether the feature survives contact with users.
A model call is a network call with opinions
It can be slow, it can fail, it can return something structurally different from yesterday, and it costs money per invocation. Every reliability pattern you already apply to third-party APIs applies here — plus a few new ones.
Force structure, then validate it
const Extracted = z.object({
title: z.string().max(80),
tags: z.array(z.string()).max(5),
confidence: z.number().min(0).max(1),
});
const raw = await model.json({ schema: Extracted, prompt });
const parsed = Extracted.safeParse(raw);
if (!parsed.success) return fallbackFromHeuristics(input);Never render model output straight into the UI. Parse it, and always have a deterministic fallback path — a heuristic, a cached previous result, or an honest empty state.
Cost control before launch, not after the invoice
- Per-user and per-day spend caps enforced server-side.
- Cache by a hash of the normalised input — repeat questions are free.
- Route simple tasks to a small model, escalate only on low confidence.
- Cap input length; a pasted PDF should be truncated, not billed.
- Log tokens per feature so you know which screen is expensive.
Design the degraded experience first
| Failure | User sees | System does |
|---|---|---|
| Timeout | Deterministic fallback result | Cancel, log, no retry storm |
| Invalid schema | Previous cached answer | One repair attempt, then fall back |
| Rate limit | Queued with a clear ETA | Backoff with jitter |
| Budget exceeded | Feature disabled with explanation | Alert, keep the app usable |
Evaluate like you test
Keep 30–50 real inputs with expected properties — not exact strings — and assert them in CI on every prompt or model change. Without an eval set, 'the new model is better' is a vibe, and regressions ship silently.
Safety basics
- Treat all model output as untrusted content; never execute or interpolate it into SQL or shell.
- Strip secrets and PII from prompts before they leave your infrastructure.
- Label AI-generated content in the interface so users can calibrate trust.
Checklist
- Structured output enforced by a schema and validated
- Deterministic fallback for every AI path
- Prompts versioned in the repository
- Timeouts with cancellation, no unbounded retries
- Per-user and per-day spend caps
- Response cache keyed by normalised input
- Evaluation set running in CI
- Model output never executed or trusted as code
- AI-generated content labelled in the UI
Common mistakes
- Shipping free-text output straight into the interface.
- Storing prompts in a database nobody reviews.
- No spend cap, discovered via the monthly bill.
- Retrying failed calls without backoff, amplifying an outage.
- Using the largest model for tasks a small one handles perfectly.
Frequently asked questions
How do I stop an LLM feature from becoming expensive?
Cache by normalised input, cap input length, route easy cases to a smaller model, and enforce per-user daily budgets server-side. Caching alone often removes a third of the traffic.
How do I test AI features?
Build a fixed evaluation set of real inputs and assert structural properties — valid schema, required fields present, no forbidden content — rather than exact text matches.
Should users know content is AI-generated?
Yes. Labelling improves trust, sets the right expectations for errors, and increasingly matters for compliance.
Summary
Constrain the output, validate it, version the prompt, budget the spend, cache the repeats, design the failure state deliberately and hold quality with an evaluation set in CI.
working on something like this?