Why Users Need To See The Work Behind The Answer
Trust in AI systems doesn’t survive a confusing or harmful decision. When a loan is denied, a recommendation misses the mark, or a resume gets filtered out, users are left with no way to understand what happened. Ability and predictability collapse in those moments, and benevolence feels irrelevant.
Explainable AI (XAI) is often treated as a data science problem, but the real challenge is designing products that translate algorithmic decisions into something humans can comprehend. UX professionals are the ones who bridge that gap. This means moving from abstract trust principles to concrete patterns for surfacing the “why” behind an AI output.
Two Practical Ways To Explain Decisions
There are many techniques for making AI more transparent, from full decision-tree logic to natural language summaries. But feature importance and counterfactuals are the two that fit most naturally into real product experiences and deliver the most value to users.
Feature Importance Shows The Key Drivers
Feature importance answers a straightforward question: “What were the most important factors the AI considered?”. The goal is to surface the top two or three variables that most influenced the outcome, not to dump the model’s entire reasoning on the user.
Example: An AI predicts whether a customer will churn. The explanation might state that “number of support calls in the last month” and “recent price increases” were the two strongest signals behind the prediction.
Counterfactuals Give Users A Path Forward
Counterfactuals answer a different question: “What would I need to change to get a different outcome?”. This gives users a sense of agency, turning a flat “no” into an actionable “not yet”.
Example: A loan applicant is denied. Instead of a blunt rejection, the system explains, “If your credit score were 50 points higher, or if your debt-to-income ratio were 10% lower, your loan would have been approved.” The user now knows exactly what to work on.
Turning Model Data Into User-Facing Explanations
UX practitioners don’t need to build the models, but understanding where explanation data comes from helps. Tools like LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations) extract the “why” from complex models, showing which inputs were most influential for a given outcome.
That raw data can be shaped into user-facing stories in three distinct ways:
- Feature importance: Answers what mattered most, using data like LIME outputs to highlight the top drivers.
- Local explanations: Answer “Why did the AI make this specific recommendation for me?”. For example, a music service might explain a song suggestion with, “Because you recently listened to several other emotional ballads and songs by female vocalists.”
- Value-based explanations (SHAP): Go a step further and show whether each factor pushed the decision toward or against the outcome. A bank loan model, for instance, might show that a high credit score pushed toward approval while a higher-than-average debt-to-income ratio pulled against it, but not enough to trigger a denial.
These layers of detail give loan officers, support agents, and end users the ability to see whether a decision was driven by positive or negative signals, not just which factors were on the table.
Good explanations aren’t an afterthought. They depend on data scientists and engineers structuring models with explainability in mind. Research and design teams should start early conversations about user needs, contribute to explainability metrics, and prototype candidate explanations collaboratively with engineers to ensure accuracy and usability.
Explainability As An Ethical Responsibility
XAI also serves a deeper purpose than building trust: it exposes algorithmic bias. Techniques like SHAP analysis can reveal when decisions are disproportionately influenced by sensitive attributes such as race, gender, or socioeconomic status, even when those variables aren’t explicit model inputs. If a loan model consistently assigns negative values to applicants from a certain demographic, that’s a bias signal that demands investigation.
The same power to illuminate also creates a risk of “explainability washing.” Like greenwashing, this happens when explanations are designed to obscure rather than clarify problematic behavior. It can take the form of overly simplified rationales that omit key factors, or framing that makes biased outcomes look neutral or fair. UX practitioners must design explanations that are verifiable and honest, not merely plausible.
Communicating a model’s limitations is part of the job too. Users need realistic expectations about accuracy, a sense of where the system is less reliable about its own performance, and clear avenues for recourse when they believe an outcome is unfair or wrong. Collaborative work among UX, data science, and ethics teams is what keeps AI systems accountable and genuinely trustworthy.
Turning Explainability Into Interface Patterns
Knowing the XAI concepts is one thing; designing them is another. A handful of repeatable patterns can translate the technical outputs of explainable AI into interface elements users actually understand.
The “Because” Statement
The most direct pattern is a plain-language statement that surfaces the primary reason for an AI’s action. The heuristic: lead with the single most impactful reason and avoid jargon.
Example: A music streaming service adds microcopy to a recommended song — “Because you listen to ‘The Fuzz’ and other psychedelic rock.” This makes the recommendation legible without requiring the user to dig into how the model works.
The “What-If” Interactive
Counterfactual explanations are inherently about empowerment, which is why the best representation is often interactive. Letting users manipulate inputs and see how the outcome changes works well in goal-oriented domains like finance or health.
Example: After a loan application is denied, instead of a dead end, the user gets a tool to explore scenarios and see how changes might alter the decision (See Figure 1).
The Highlight Reel
When an AI acts on user content — summarizing a document or tagging faces in photos — the explanation should be visually connected to the source material. Highlighting, outlines, and annotations tie the explanation to the specific interface element it describes.
Example: An AI summarizer displays a generated point (“Initial research showed a market gap for sustainable products”) alongside the highlighted sentence in the source document that supports that claim. This links the summary back to its evidence in the original text.
The Push-and-Pull Visual
Complex decisions often involve multiple competing factors. Simple, color-coded data visualizations can show the interplay of positive and negative influences without overwhelming the user.
Example: An AI screening tool explains why a candidate is a 75% match by listing factors that push the score up (5+ years UX research experience, proficient in Python) versus those that pull it down (no B2B SaaS experience). The visual separation makes the reasoning legible at a glance.
Beyond the Core Patterns
Additional techniques can complement these patterns:
- Natural language explanations: Translating technical AI output into conversational language non-experts can grasp.
- Contextual explanations: Offering a rationale at the precise moment and location it is most relevant to the user’s task.
- Relevant visualizations: Using charts, graphs, or heatmaps to make the AI’s decision process intuitive.
A note for the front end: Implementing these patterns introduces its own technical concerns. Developers must design APIs that efficiently retrieve explanation data, and real-time generation of explanations for every user interaction requires careful planning to avoid latency.
XAI in Practice
Three examples illustrate how these principles play out in real products — and what happens when they are neglected.
UPS Capital’s DeliveryDefense
UPS assigns a “delivery confidence score” to each address, predicting the likelihood of package theft from historical loss data and other factors. When an address scores low, the system reroutes the package to a secure Access Point and explains the action — for instance, noting a history of theft at the original address. The transparency here serves both risk mitigation and customer trust.
Autonomous Vehicles
Self-driving cars must be able to explain their behavior in real time. When a vehicle brakes suddenly, a system can identify the cause — for example, a pedestrian stepping into the road. This explanation matters for passenger comfort and regulatory accountability alike.
IBM Watson Health
Watson for Oncology is frequently cited as a cautionary case for XAI. Its recommendations sometimes conflicted with local clinical practice because they derived from U.S.-centric guidelines, and its explanations were not always clear. The failure underscores why robust, context-aware explainability matters in safety-critical fields.
Researching What to Explain
Explanations only help when they answer questions users actually have. An explanation that addresses a question nobody is asking is just noise. That makes UX research the connective tissue in an XAI strategy: it identifies where explanations are needed and validates whether the designs that deliver them work.
Mental Model Interviews
Before designing any explanation, researchers need to understand how users perceive the AI system. Semi-structured interviews can reveal these perceptions by prompting users to describe — even draw — how they believe the AI works: its inputs, outputs, and internal logic.
These conversations frequently expose misconceptions. A user may assume a recommendation engine uses only their viewing history, unaware that the algorithm also factors in time of day, trending content, or the behavior of similar users. Surfacing that gap between the user’s mental model and the system’s actual logic tells researchers precisely what needs to be communicated to build a more accurate understanding — a fundamental step toward earning trust.
AI Journey Mapping
Mapping the user journey with an AI feature identifies the exact moments where confusion, frustration, or distrust emerge. A playlist recommendation that feels random shakes confidence in the system’s curation. An AI photo tagger misidentifying a family member is more than a glitch; it undermines perceived accuracy and personalization.
These pain points signal where a well-placed explanation can repair a breach of trust before it leads to abandonment. Journey mapping pushes the focus beyond explaining the final output toward explaining the process at critical junctures. Useful explanations address:
- Why a particular output was generated — whether due to specific input data or model architecture.
- Which factors influenced the decision, and how they were weighted.
- How the AI arrived at its conclusion, in simplified terms.
- What assumptions the AI made about the user’s intent or data.
- What the AI’s limitations are, setting realistic expectations.
This technique turns XAI from an abstract concept into an actionable framework, pinpointing where trust is at stake. Research may also reshape the strategy entirely: a team debating how to explain a loan denial might discover users are actually more concerned with how their data was used in the first place. Without research, teams are guessing at what users are wondering.
Validating Explanation Designs
Once research has identified what to explain, the collaborative loop with design begins. Designers prototype the patterns described above — the “Because” statement, the interactive sliders — and researchers test whether those prototypes communicate effectively.
Targeted usability and comprehension testing: Studies should test the XAI components specifically rather than overall usability. Instead of asking only “Is this easy to use?”, researchers ask users to explain in their own words why the system recommended a product, or to demonstrate how they would change the outcome. The goal is measuring comprehension and actionability, not just usability.
Measuring trust directly: Simple surveys using rating scales can be administered before and after an explanation is shown. Asking users to rate their trust in a recommendation on a 5-point scale both before and after they see the “Because” statement provides quantitative data on whether the explanation moves trust at all.
The result is a continuous loop: research informs initial design, testing feeds findings back to the design team for refinement, and revised explanations are tested again. Whether the problem is jargon in the “Because” statement or a “What-If” slider that confuses more than it empowers, this iterative process ensures the final explanations are technically accurate, understandable, and genuinely useful.
Explanations That Fit: The Art of Right-Sized Detail
More explanation is not always better. Presenting every model variable to a user creates cognitive overload and can actually undermine trust. The goal is to make the user confident, not to make them a data scientist. The key is to find the “just right” level of detail for each person and context.
The most effective way to achieve this is through progressive disclosure, a layered approach that respects both attention and expertise:
- Start with the simple. Lead with a concise “Because” statement that most users will find sufficient.
- Offer a path to detail. Include a low-friction link like “Learn More” or “See how this was determined.”
- Reveal the complexity. Behind that link, provide the deeper exploration tools: interactive sliders, visualizations, or a detailed list of contributing factors.
Consider a smart home device that recommends optimal heating settings. The initial explanation might be: “Your home is currently heated to 72 degrees, which is the optimal temperature for energy savings and comfort.” Below that, a small link asks: “Why is 72 degrees optimal?” Clicking it opens a screen showing a more detailed picture that might include:
- Interactive sliders for outside temperature, humidity, and your preferred comfort level, showing how these adjust the recommended temperature.
- A visualization of energy consumption at different temperatures.
- A list of contributing factors like “Time of day,” “Current outside temperature,” “Historical energy usage,” and “Occupancy sensors.”
This layered approach naturally encourages the combination of multiple XAI methods. You can begin with a simple feature-importance statement for immediate understanding, then provide a “Learn More” link that unveils a counterfactual “What-If” tool or a value-based “Push-and-Pull” visual for deeper examination.
A concrete example: a loan application system could state the primary reason for denial, then let the user manipulate their income or debt figures in a “What-If” tool to see how those changes would alter the outcome. Finally, a detailed chart could illustrate the positive and negative contributions of all factors. Users access the level of detail they need, when they need it, without overwhelming anyone.
Choosing Techniques Through Research
Selecting the right XAI toolkit is fundamentally a UX research question, not a technical one. Mental model interviews and AI journey mapping are essential for understanding user needs and pain points around AI trust. Mental model interviews uncover misconceptions about how the AI works, indicating where baseline explanations (like feature importance or local explanations) are needed. AI journey mapping highlights critical moments of confusion or distrust, signaling where more granular or interactive explanations would best rebuild confidence and provide user agency.
The optimal choice of technique is always the one grounded in actual user research — explanations should address real questions and concerns, not merely present technical details for their own sake.
Curation for Deep Reasoning Agents
Newer AI systems classified as deep reasoning agents generate an explicit logical chain of thought for each complex task they handle. Rather than simply citing sources, they show the step-by-step path taken to arrive at a conclusion. The transparency is valuable, but a full play-by-play spanning multiple paragraphs can feel overwhelming to a user who just wants to complete their task.
The Goldilocks Zone principle applies here too. Curate the journey by showing only the final conclusion and the most salient step in the reasoning first. Users who need to verify the logic or find a specific fact can opt in to view the complete, detailed multi-step reasoning. This approach preserves full transparency while respecting user attention.
Advancing XAI in Your Organization
Explainability is a foundational pillar of trustworthy and effective AI products. For practitioners aiming to drive organizational change, the work extends beyond design patterns into advocacy and continuous learning. Useful resources to explore include the AI Explainability 360 (AIX360) toolkit from IBM Research and Google’s What-If Tool, both of which offer hands-on ways to explore model behavior. Communities like the Responsible AI Forum and human-centered AI research groups offer collaboration and insight opportunities.
Frame explainability as a strategic investment when pitching to leadership or cross-functional stakeholders:
“By investing in XAI, we’ll go beyond building trust; we’ll accelerate user adoption, reduce support costs by empowering users with understanding, and mitigate significant ethical and regulatory risks by exposing potential biases. This is good design and smart business.”
Practical understanding and a clear voice are essential for moving AI out of the black box and into a collaborative partnership with its users.
