Take a health insurer whose appeals specialists are handed an AI that drafts the full written determination. The system behaves exactly as specified, yet specialists abandon it — and the few who persist take longer per case than colleagues who ignore it. A short evaluation explains why: vetting a machine-written determination closely enough to sign it costs roughly the same as writing one, and the reviewer must also hunt for mistakes they would never have made. Rescoping fixes it. The second version surfaces the policy provisions and prior decisions tied to each appeal and leaves the writing to the human; the search step vanishes, review time falls, and the specialist keeps the part of the job they excel at. Same technology, different division of labor.
In another setting, a fraud screening tool clears the bar in the opposite direction: analysts review flags they would never have raised themselves, paying attention for a catch rate that repays it. The deciding factor in both cases is whether the return beats the cost of adjusting, for one user doing one task — a threshold we can call the break-even line.
Two Sides Of The Ledger
Anyone facing an interface change pays before they gain. They rebuild their mental picture of the tool, relearn where things live, and settle how far to trust the output. Nelson Cowan's 2001 review of working memory research puts the ceiling at roughly four items held simultaneously, and all three of those jobs draw from that same supply.
AI raises the stakes beyond a menu redesign because it reshapes the working day. Delegate synthesis and the user becomes a checker; delegate the recommendation too and the role narrows to sign-off. Either shift drains attention, and for people whose training underwrites their standing, either can read as demotion. In a study by Sophia Ackerhans (2025) and colleagues, physicians asked to sign AI-assisted decisions reported a heightened threat to professional identity — and explanations of the AI's reasoning lifted trust and that threat in tandem.
On the return side, three forms count: removed effort, errors caught that the person would have missed, and consistency across people doing the same work.
An AI feature earns its place when what it returns beats what it costs the user to adjust, for one user doing one task.
Running The Audit
Four questions locate the break-even line before design begins, each yielding a design condition the brief can act on. They do not settle the build decision — funding, strategy and existing commitments weigh there too, and four questions cannot see any of it. Use them as a set, while the feature is still a proposal, and involve at least one person who does the task for a living: a loan officer, an analyst, a benefits specialist. A brief conversation reveals where the mental effort really sits. A roadmap seat is not required.
1. Does this shrink the total effort for this user?
Count everything — thinking, clicking, and the checking AI will owe. A lower click count proves nothing by itself. Log the cost even when it rises; a later question may uncover a return that justifies it.
Consider a loan officer reading 40 financial disclosures to isolate the three that conflict with lending policy: heavy thinking, which AI that surfaces those conflicts cuts directly. Now consider typing a loan number into a tracking system — seconds, almost no thought. Pre-filling that field moves the cost rather than removing it, because the officer now verifies the entry. Here is where the standard advice to automate the repetitive and monotonous misfires:
- An AI-filled field transfers checking duty to the user.
- The original task needs a short keystroke sequence and no sustained attention.
- Automation swaps a low-effort action for a high-attention one.
Compare total effort before and after — counting inherited checking — to find what is genuinely worth automating.
Who the user is matters as much as the task. A junior analyst handed an AI-drafted risk summary gains a starting point they lack the pattern library to produce; a senior analyst reads the same draft against an analysis already formed, spends four minutes, and learns nothing. That finding identifies the segment to ship to: scope the addition to the group it serves.
Design condition: Name the user segment the addition serves, and design an exit for the other segments.
2. Does this beat the user's current method?
Benchmark against what the user does now, spreadsheets and workarounds included. AI wins where genuine ambiguity exists — exploring options, producing a starting point, finding patterns too numerous to spot by hand — and also on scale, on speed, on consistency across a group, and on catching what the current method reliably misses. That last one is how fraud screening earns its place. Once the user has a practiced route to a known answer, the advantage evaporates.
An HR team building succession planning sees both answers inside one product. Surfacing internal candidates whose skill combinations nobody would think to search for wins on ambiguity and scale together: no single right answer exists, and manual review of that search space would take weeks. Drafting the notification email after a candidate is picked loses — the format is familiar, writing takes two minutes, and the sender holds firm opinions about tone. The search earns the addition; the email draft layers an interaction onto a task that runs fine without one.
Input nature and output shape are separate calls; a task can be genuinely ambiguous and still call for a structured visual result rather than a conversational one. Pair this audit with a modality evaluation for the output side.
Design condition: Remove AI from any step where the user's current method already reaches the answer quickly and reliably.
3. Can the user catch the errors?
Every AI system errs; the open question is whether your user notices. Weigh how visible the error is, what it costs when it slips through, and whether the user retains the skill to catch it. Where Question 1 prices checking, this one asks whether checking is even possible.
Visibility varies widely and teams estimate it badly. An AI trip planner proposing a museum on its closing day produces an invisible error — nothing in the output looks wrong, and the traveler discovers it outside a locked door in Tokyo. A suggested flight departing before the connecting flight lands shows the contradiction on screen.
Automation bias worsens the invisible cases. Raja Parasuraman and Dietrich Manzey (2010), reviewing decades of human factors research across aviation, medicine and process control, found that people defer to automated output even when it is wrong. Trust climbs fast and rarely by conscious choice; once it does, users stop applying the scrutiny that would catch the mistake.
Map the error surface — the ways this feature can be wrong — before design starts. List the three likeliest error types and rate each on visibility and consequence. Any low-visibility, high-consequence pairing becomes a design requirement.
Show users where the output came from and how it was produced. That is provenance, and it settles trust at the right level. Someone told a restaurant's hours came from a page cached in March knows what to check; a confident paragraph with no source leaves them guessing.
Design condition: Any addition with a high-consequence, low-visibility error surface ships with provenance display and verification prompts, or it does not ship.
4. After a year of use, can the user still tell when the AI is wrong?
Ceding routine work is a genuine benefit. Nobody needs to preserve a specialist's ability to format a citation or an analyst's talent for totalling a column by hand. The judgment that catches the machine's mistakes is exempt, though: it makes every other delegation safe, and disuse degrades it like any skill.
Radiology research shows how quickly a suggestion overrides a reader's own judgment. In a 2023 experiment, Thomas Dratsch and colleagues had 27 radiologists read mammograms alongside what they were told was an AI system (a simulation built solely for the study). After a 10-image training set in which the suggested category was always correct, 40 more mammograms followed with incorrect suggestions planted in 12. Incorrect suggestions damaged diagnostic accuracy at every experience level. The most experienced readers, averaging 15 years, assigned the correct category 82.3% of the time with a right suggestion and 45.5% with a wrong one; the least experienced fell from 79.7% to 19.8%. Experience softened the effect without preventing it. A year of deferring to one system, with the replaced judgment unused, leaves a reader weaker than these radiologists started.
Question 3 examines the user today; stretching the same test 12 months out gives Question 4. Teams that answer the first and skip the second misread the resulting failures as model quality problems, when the cause is human judgment taken out of circulation.
Ask whether this user could judge output quality 12 months from now if the feature vanished tomorrow. If not, the design needs a mechanism keeping verification skill in use: periodic blind review (judgment formed before seeing the AI output) costs little, and rotating a share of cases off assistance or a weekly machine-versus-human comparison do similar work.
Design condition: Name the mechanism that keeps verification skill in use, and build it into the workflow itself. A slide in the training deck will not do it.
Reading The Answers Together
Table 1 gathers the four questions, the signals pointing each way, and the condition each produces.
| Audit question | AI earns its place when… | AI costs the user when… | Design condition |
|---|---|---|---|
| Does this reduce the total effort the task takes for this user? | The task demands synthesis across large or varied inputs, and judgment calls create the bottleneck | The task follows a known procedure with a fast manual path, or verification costs as much as the original work | Name the segment served, and give other segments an exit |
| Does this beat the way the user does the task now? | The AI approach wins on ambiguity, scale, speed, consistency, or catch rate against the method the user has now | The user’s current method already reaches the answer quickly and reliably | Remove AI from steps the current method already handles well |
| Can the user catch the errors? | Errors appear on their face, stakes stay low, correction is cheap | Errors stay subtle, stakes run high, or verification needs the skill being delegated | Provenance display and verification prompts, or no build |
| After a year, can the user still tell when the AI is wrong? | AI absorbs volume while the user keeps making the judgment calls | AI absorbs the judgment calls with no feedback on quality | A named mechanism that keeps verification skill in use |
Table 1: The AI Addition Decision Matrix. Clear answers on all four questions point toward building as proposed. A poor answer on Question 1 calls for a design condition or a scope change — unless Question 2 finds a return covering the cost. Questions 2 and 4 follow the same pattern without that exception, and a poor answer on Question 3 can stop the build outright.
Write the conditions into the design brief, where they become project requirements. Roadmap reviews push teams to reduce AI features to yes or no, and that discards everything the audit found. A feature strong on Questions 1 and 2 but weak on Question 3 proceeds only with provenance built in — on day one. Requirements arriving at the end read as polish, and polish is what gets cut when timelines slip.
Applying the four questions to the appeals tool
Version one of the insurer's appeals tool stumbles on Question 1. A determination calls for sustained thought, and drafting it with a model merely relocates that effort into verification. An hour spent with the questions would have caught this before design.
Question 2 agrees. Appeals determinations follow a familiar structure, and specialists reach the answer along a practiced route. What is genuinely ambiguous sits upstream: locating the provisions and prior decisions that apply.
Question 3 is where the stakes show up. Someone's medical coverage depends on the determination, and an error wrapped in fluent prose stays hidden until the next appeal arrives. That is the reasoning behind the second version's provenance requirement: every provision it surfaces carries its source document and retrieval date.
Question 4 is what makes version two durable. Specialists still author each determination, so the judgment that weighs retrieved material stays exercised daily. Had version one stayed, the team would have spent a year training specialists to approve text they could no longer properly assess.
Running the same audit on meeting summaries
Meeting summaries generated by AI reach millions of users daily, and the four questions point somewhere else for them.
Under Question 1, a summary eliminates note-taking but introduces the task of verifying whether a decision was pinned on the right person. Aggregate effort falls for the participant who never took notes; for the person who habitually took them and circulated them, the trade is roughly even, and the checking burden lands there. Surfacing exactly that split is the point of the question.
Question 2 sets the summary against a colleague typing while half-listening, which yields patchy notes. On completeness, and on consistency from one meeting to the next, the summary wins, so most teams clear this question.
The dangerous failure mode is a summary that credits a commitment to the wrong person while reading exactly like a correct one — no one goes back to the transcript. Under Question 3, low visibility combined with real consequence becomes a requirement: every action item needs a linked source timestamp.
Question 4 clears with no mechanism needed. Giving up meeting notes costs nobody a skill worth protecting, and the participants continue to make each decision the summary records.
The verdict: two clear passes, one segment finding, one condition. The timestamps ship, and the habitual note-taker is the segment to watch after launch.
Auditing in 60 minutes
This can run before the next AI feature is added to the roadmap.
- Specify the task (10 minutes). State it for one user in one sitting — for example, help a mid-level financial analyst identify the three highest-risk items in a 40-page due diligence report during a two-hour review window. "Helps users summarize documents" is too vague to evaluate.
- Answer the questions with a practitioner (25 minutes). Whoever does the work knows where the effort actually sits better than any product estimate. Ask what they find hard, watch whether the proposed feature touches those parts, and ask what they would lose the ability to check if the feature made the judgment calls for a year.
- Chart the error surface (15 minutes). Name the three most likely error types and rate each on visibility and consequence. Low visibility plus high consequence becomes a build requirement: a diagnostic tool suggesting a treatment plan that carries high mortality risk if wrong demands strict provenance and verification triggers, while a scheduler proposing the wrong meeting time is spotted and fixed easily.
- Put the conditions in the brief (10 minutes). Provenance display, verification prompts, segment scoping and skill mechanisms belong in the same design document as the feature description.
Everett Rogers (2003) named this pro-innovation bias: treating new capability as progress before anyone has verified the benefit, with competitive pressure converting a rival's announcement into an internal mandate. An hour of structured questions is how a team answers that pressure.
Taking the findings into the product review
Audit results misfire when a designer presents them as grounds to kill the feature. The conditions are the deliverable, so carry those into the review.
Begin with the segment the feature serves, which turns the answer into a yes for somebody. Then hand over the error surface as a requirement with a price attached — a product manager can act on timestamps behind every attributed decision; no one can act on a vague worry about trust. Finally, offer the reduced scope unasked. A version that surfaces the provisions rather than writing the determination is a smaller build with a stronger adoption story, and that framing moves the roadmap discussion forward.
A sentence that works spoken aloud: We think this clears the bar for junior analysts and adds a verification step for senior analysts, so we would scope it to onboarding and revisit the rest next quarter.
No matter how capable the model becomes, the attention available in a working day stays fixed. Before the next feature reaches design, record what it costs the person doing the work and which of the three returns pays for it. With both in hand, the team walks into the roadmap review with a scope to propose.
References
- “Perceived trust and professional identity threat in AI-based clinical decision support systems: Scenario-based experimental study on AI process design features”, Ackerhans, S., Wehkamp, K., Petzina, R., Dumitrescu, D., & Schultz, C. (2025)
- “The magical number 4 in short-term memory: A reconsideration of mental storage capacity”, Cowan, N. (2001)
- “Automation bias in mammography: The impact of artificial intelligence BI-RADS suggestions on reader performance”, Dratsch, T., Chen, X., Rezazade Mehrizi, M., Kloeckner, R., Mähringer-Kunz, A., Püsken, M., Baeßler, B., Sauer, S., Maintz, D., & Pinto dos Santos, D. (2023)
- “Complacency and bias in human use of automation: An attentional integration”, Parasuraman, R., & Manzey, D. H. (2010)
- “Diffusion of Innovations”, Rogers, E. M. (2003)




