AI8 min read
Designing trust into AI products
Trust in an AI product is not built by being right. It is built by being checkable, correctable and honest about the edges.
By Strategy Practice, Brand, strategy & AI
Teams building AI features tend to treat trust as a function of accuracy. Get the model good enough and trust follows. In practice the relationship is much weaker than that, and the gap is where most adoption failures live.
We have watched capable models sit unused, and less capable ones get adopted enthusiastically. The difference was never accuracy. It was whether a user could check the output cheaply, correct it cheaply, and tell in advance which cases the system was weak on.
The asymmetry that governs everything
Trust is built slowly by correct outputs and destroyed quickly by confident wrong ones. A system that is right 95% of the time and cannot indicate which 5% is often less useful than one that is right 85% of the time and flags its uncertainty accurately.
That is not a modelling observation. It is a design observation, and it has a design response: calibrated disclosure.
Provenance before conclusion
The single highest-value pattern in professional AI products. Every finding cites what it rests on, the document, the page, the passage, and clicking it opens the source in place.
Provenance is usually justified as a trust feature. It is better understood as an error-detection feature. When users can see what a claim rests on, they catch errors at a glance rather than by re-reading everything. A finding with a weak or missing citation becomes visibly suspicious, which is information the interface could not otherwise convey.
Express uncertainty in language, not percentages
Confidence percentages are close to useless in an interface. Users cannot calibrate them, they are frequently not calibrated by the system either, and they collapse different kinds of uncertainty into one number.
| Instead of | Say |
|---|---|
| 73% confident | Uncertain, the policy wording is ambiguous on flood exclusion |
| Low confidence | Only one source supports this; usually there are three |
| Unable to process | Pages 4 to 7 are scanned images I could not read |
| Verified | Matches the contract and the amendment from March |
Each of these tells the user where to look. A number tells them how to feel.
Correction has to be the cheapest path
If disagreeing with the system is slower than working around it, everyone works around it. The product then loses the only real signal it has about its own failure modes, and the team ships improvements aimed at problems users stopped reporting.
- Correction inline, on the specific finding, not as a separate feedback form
- The reason captured as structured data, not free text nobody reads
- Visible acknowledgement that the correction was received and will be used
- Corrections routed to the team that can act on them, with volume reported
Match autonomy to consequence
A single autonomy setting across a product means it is either too cautious to be useful on routine work or too aggressive on consequential work. Tier it explicitly, and show the user which tier they are in.
Tiered autonomy
- Assist: organises and cites, concludes nothing. For high-consequence cases.
- Recommend: proposes with evidence and a stated reason for uncertainty.
- Draft & route: acts on well-precedented, low-consequence, reversible work, with sampling.
Be honest about the edges
Products that state their limits plainly are trusted more than products that imply generality. 'I can read text-based PDFs; scanned documents need OCR first' sets an accurate expectation and prevents the silent failure that would otherwise cost you a user.
Users do not need a system that is always right. They need a system whose wrongness is predictable, visible and cheap to fix.
Strategy Practice
Brand, strategy & AI
The strategy practice covers positioning, product strategy, UX diagnosis and AI opportunity work. It writes about the commercial reasoning behind design decisions, and the organisational conditions that make good products possible.
