By Mehdi HassanMost AI features fail for one of three reasons: nobody wanted them, the output wasn't useful, or the model couldn't produce good output reliably. A normal clickable mockup can't tell you which one you're facing, because the hard part of an AI feature is what it says, not where the button sits. This guide covers the three methods we use to test AI features before committing to a build, and it sits inside our broader guide to prototyping for product teams.
Why are AI features harder to prototype than normal features?
AI features are harder to prototype because the output is variable. A static Figma screen shows one perfect answer, but real users will get a range of answers, some excellent, some wrong and some confidently wrong. A test built on the perfect answer measures your best case, not your product.
Two more variables never show up in a mockup: latency, which changes how the feature feels, and cost per use, which decides whether it fits your pricing. So we treat AI prototyping as a sequence of small tests, each aimed at one unknown, rather than one big demo.
How do you prototype AI features before building them?
You prototype AI features by running the cheapest test that answers your riskiest question, then moving to a more expensive test only when the cheaper one passes. In practice that usually means demand first, usefulness second and model capability third, though a technically risky feature can flip the order.
Define the job and the success moment
Write one sentence for what the user is trying to get done and what a good result looks like to them, in their words. "Draft a reply I'd send with light edits" is testable. "AI-powered inbox" isn't.
Pick the riskiest unknown
Demand, usefulness or capability. Rank them honestly. Founders usually overrate demand risk and underrate capability risk.
Run the matching test
Fake door for demand, Wizard of Oz for usefulness, a thin real-model prototype for capability. Keep each test small enough to finish inside a week.
Design the failure states
Include a wrong answer, a slow answer and a refusal in every prototype, so you see how users react when the AI isn't perfect.
Decide and document
Write down what you learned, what you'd build and what you'd cut. Hand that to whoever builds it, along with the prompts and example outputs.
If you need the whole team aligned on the problem first, a design sprint is a good wrapper: map the problem early in the week, then use one of the methods below as the prototype and test it with users at the end.
What is Wizard of Oz testing for AI features?
Wizard of Oz testing is a method where a person secretly produces the output that the AI would produce, while the participant uses a realistic interface. It tests whether the output is useful before anyone builds a model pipeline. The name comes from the man behind the curtain.
For an AI feature, the setup is simple. Build a front end that looks real, often a chat panel or a "Generate" button in a clickable prototype. When the participant submits, a teammate writes the response by hand or edits a model draft, then sends it back. You learn what users ask for, which outputs they act on and which they ignore.
Pros
- Tests the value of the output before any engineering
- Lets you try several answer formats (bullet list, draft, table) in one session
- Shows what users actually type, which becomes your first prompt and eval set
Cons
- A human is smarter and slower than a model, so it can flatter the concept
- Doesn't scale past a handful of sessions
- Needs a debrief: tell participants afterwards that a person wrote the answers
To counter the flattery, have the wizard lightly edit a model draft and include one weak answer per session. That keeps the test closer to what a real model will deliver.
When does a fake door test make sense for an AI feature?
A fake door test makes sense when your biggest unknown is whether people want the feature at all. You add an entry point, such as a button, menu item or banner, for a feature that doesn't exist yet, then count who clicks and show an honest message explaining it's coming.
It's the cheapest test of the three because it needs no AI at all. The trade-off is that it only measures curiosity. A click on "Summarize this thread" tells you the label is appealing, not that the summaries will be good. Use it to choose between candidate features, not to validate one.
On a marketing page, a fake door is just a split test; see A/B testing in Framer, Webflow and Next.js.
When should you prototype with a real model?
You should prototype with a real model when the question is capability: can a model produce good enough output on your users' real, messy inputs, fast enough and cheaply enough. No mockup or human wizard can answer that.
Start in a model provider's playground with a few dozen real examples, then write a short prompt and judge the outputs against your success sentence. If that passes, build a thin code prototype: one screen, a real API call, streaming output and a way to rate each answer. That's where you'll see latency, odd failures and how long prompts get. Our guide to building AI features in a web app covers streaming, retrieval and cost control once you move past the prototype.
This is also the point where a code prototype beats a Figma prototype. A Figma file can't stream tokens or show a real refusal. A small Next.js page can, and parts of it often survive into production.
For example, suppose you're testing an AI meeting-notes feature
For example, suppose a B2B tool wants AI meeting summaries. A fake door ("Summarize this call") checks whether people reach for it. A Wizard of Oz round with summaries in three formats shows users want action items with owners, not a paragraph. A real-model prototype on consented transcripts then reveals that names get mixed up when two speakers share a first name, so the design gains a speaker-confirmation step.
This is a hypothetical walkthrough to show the sequence, not a real project.
What does each AI prototyping approach cost?
Each approach costs mostly team time, and the order from cheapest to most expensive is fake door, Wizard of Oz, then real model. The real-model prototype is the only one with usage costs, and it's also the only one that tells you whether the feature works.
Approach | Answers | Main cost | Relative effort | Misses |
|---|---|---|---|---|
Fake door test | Do people want it? | A button, a message, analytics | Lowest | Output quality, feasibility |
Wizard of Oz | Is the output useful? | Prototype plus a person in every session | Low to medium | Latency, scale, real error rate |
Playground test | Can a model do it at all? | Engineer or designer time, small API usage | Low | UX, real user behaviour |
Thin code prototype | Does it work for real users? | Build time, API usage, test sessions | Medium to high | Long-term quality drift |
The expensive mistake is skipping straight to a polished production build, because you then pay for the full pipeline before learning the output wasn't wanted. If you want a team to run these tests with you, our $3,000 brand and design sprint can cover the interface explorations for a new AI feature, and a $1,000 trial week of design and development is a low-commitment start on a first code prototype.
What should you measure when you test an AI prototype?
Measure what users do with the output, not what they say about it. The strongest signal is whether they use the answer as is, edit it lightly, rewrite it or ignore it. Ask after the task, not during it.
- Acceptance: how often the output gets used with no or light edits.
- Recovery: what users do after a wrong answer. Do they retry, rephrase, or give up on the feature?
- Trust cues: whether they look for a source, a confidence signal or a way to undo.
- Wait tolerance: where in the response they lose patience, which shapes your loading and streaming design.
- Inputs: the real prompts and files people use, which become your test set for the build.
These signals feed straight into interface decisions covered in our guides to AI product UX and AI chat interface design. If the feature takes actions on the user's behalf rather than just answering, test approval and undo flows too, which our piece on agentic UX covers in depth.
Which prototyping tools work best for AI features?
The best tools depend on the method. Fake doors need only your existing app or site plus analytics. Wizard of Oz tests work well in Figma or Framer prototypes with a hidden channel for the wizard. Real-model tests need a provider playground first, then code.
AI app builders can produce a working chat interface quickly, which helps a capability test, but they tend to hide the prompt and error handling you need to inspect. We compare options in the best prototyping tools in 2026. If the prototype's real job is fundraising, read our guide to investor demo prototypes.
