Rethinking How Voice Interfaces Deliver Feedback
Voice assistants dominate the public perception of voice user interfaces, and they are, for most people, the only context in which they have spoken to a computer. But the assistant model — where the system speaks back in natural language — is not the only way to design voice interactions, nor is it the best fit for many uses.
This approach comes with significant constraints. Relying on spoken responses creates two structural problems that limit the usefulness of voice UIs far beyond basic commands like setting timers or playing music. An alternative is to use voice only for input and rely on visual, graphical feedback for output — a model that sidesteps these limitations entirely.
What Defines a Voice Assistant
A voice assistant is fundamentally a voice-based chatbot. It uses natural language in place of icons and menus, and it applies that language in both directions: the user speaks to the system, and the system replies in spoken language as well. This conversational design creates an experience resembling a dialogue with another person.
This is not a new concept. Early dialogue systems like Microsoft Clippy attempted to guide users through office software based on what it inferred they were trying to do. Today, chatbots in customer support and general-purpose assistants like Google Assistant or Amazon Alexa follow the same pattern, answering questions and offering proactive help. As the underlying technology improves, these conversations can only grow more open-ended.
Two defining traits emerge from this design. First, the interface uses conversation instead of transactions. Second, the tone and language of the system craft a persona — a sense of empathy and service akin to dealing with a real person.
That illusion of humanity is also where the model runs into trouble.
The Trouble with Spoken Responses
Voice assistants use speech for both input and output. While that makes input fast and natural, the spoken output causes problems that are not easy to engineer away.
False Expectations of a Human
Humans instinctively project human characteristics onto objects — a phenomenon called anthropomorphism. A graphical interface can feel relatively neutral, but a spoken voice immediately invites assumptions about age, gender, and personality. The user begins treating the system as if it were human.
Yet humans are highly sensitive to things that imitate them imperfectly. The closer a voice gets to sounding natural, the more its slips bother us. Voice assistants fall squarely into the “uncanny valley.” When a system sounds human enough to raise expectations, even small errors feel idiotic or even offensive, and the user experience sours.
This is the core promise of the assistant paradigm — a human-like service experience that infallibly, inevitably fails. So far, no system has passed the Turing test, and the assistant model promises a level of conversational intelligence it cannot deliver. The more trust a user places in an assistant, the more disappointing its inevitable missteps become.
Sequential, Sluggish Interaction
Spoken dialogue also slows down the transaction. Natural language responses force a strictly turn-based exchange, and this is by design of our own cognition.
The human brain uses two distinct processing systems:
- Linguistic system: processes speech.
- Visuospatial system: processes what we see and spatial relationships.
These systems run in parallel, but each can handle only one stimulus at a time. That is why talking while driving works but texting while driving does not — both texting and driving demand the visuospatial channel. The same logic dooms spoken dialogue with voice assistants, where one participant holds the conversation alone and the other listens passively.
Compare this to a face-to-face discussion of a complex subject, which works best in person precisely because both parties can engage simultaneously, using visual cues alongside speech. Assistants are reduced to unidirectional exchanges because they rely on natural language for output. They use endpointing to decide when the user has finished talking, and remain deaf while they beremain speaking themselves.
While voice is up to four times faster than typing for input, it is far slower than reading for absorbing the response. A fully spoken interaction works well for short commands — “turn off the lights” — where very little feedback is required. But for anything more elaborate, requiring the user to listen to a sequential, spoken output is a poor use of the medium.
Voice for Input, Visuals for Output
An alternative approach uses voice only to take input from the user, while the system’s output remains visual through a more traditional graphical interface. This gets the best of both worlds: voice input is natural, fast, and effortless, while the visual feedback avoids the uncanny valley and eliminates the sequential bottleneck of prosod responses. Information can be scanned, compared, or skimmed at a glance, and the UI stays as responsive as any visual interface.
Visual Feedback Instead of Conversation
Replacing a voice assistant’s natural language responses with realtime visual feedback solves several of its core problems. When the interface shows what it understands as the user speaks, the user and system can exchange information simultaneously rather than in a strict turn-taking loop. That bidirectional flow carries far more information than a spoken exchange, and the UI never has to interrupt the user to confirm or clarify.
Today’s voice assistant use cases — alarms, music, weather, simple questions — are all low-stakes tasks. Failure is annoying but not costly. The jobs people actually want help with, as David Pierce of the Wall Street Journal put it, are different:
“I can’t imagine booking a flight or managing my budget through a voice assistant, or tracking my diet by shouting ingredients at my speaker.”
— David Pierce, Wall Street Journal
Those are information-heavy tasks that need to go right. And while any voice UI will eventually make mistakes, the real problem with assistants is how slowly errors surface and how awkward recovery is. Typing errors are trivial to fix — one backspace and retry. A misheard conversational command forces the user into a strange dialogue with a virtual persona. Fast recovery from errors is what keeps users efficient and frustration low.
Direct Voice Interaction
Rather than asking an avatar to perform a task, a user can manipulate the graphical interface directly with voice. This is direct voice interaction: voice acts as another modality alongside touch, click, and swipe for operating the same widgets the user already sees.
This may sound like a semantic distinction, but it is critical. Clicking a button makes it obvious you are operating a machine; adding voice to that same button simply improves the human-computer interaction. The assistant paradigm, by contrast, tries to simulate human-to-human interaction — and lands in an uncanny valley where the simulation is worse than the real thing.
Direct voice interaction also lets each modality do what it does best. Voice is excellent for inputting rich information, while touch is better for selecting among a few visible alternatives. A user can say, “Show me flights from London to New York departing tomorrow,” then tap the best option from the resulting list. They can switch between voice and touch seamlessly, based on context and task.
Given the benefits, why haven’t the major tech companies shipped this? One reason is that the assistant model suits how they monetize user data. Another is architectural: speech recognition, turning audio into text, has traditionally run separately from natural language understanding, which extracts intent — recognizably the same action from “turn off the living room lights” and “please switch off the lights in the living room.” In an assistant, those two steps run sequentially, which is why there is a pause after you finish speaking before the action happens. The transcript appears in near realtime; the action waits.
Streaming Spoken Language Understanding
An interface feels responsive when it reacts to input the moment that input is meaningful. The iPhone’s breakthrough was a touchscreen that responded instantly; voice needs the same. That requires streaming spoken language understanding, which processes intent from the first word spoken instead of waiting for a pause.
The user gets immediate confirmation the system understands, mid-sentence if needed. This acts like a nod or “a-ha” in conversation, encouraging longer and more complex utterances. It also offers fast recovery when the user misspeaks: the mistake is visible right away, and the user can correct naturally — “I want this, no, I meant that.” This is more like fixing a typo than explaining yourself to a confused assistant. The whole exchange becomes faster because the user is not limited to the roughly 150 words per minute of spoken response; visual feedback allows much higher information density.
Toward Frictionless Interfaces
The near-universal adoption of the assistant paradigm has obscured what voice is actually good at: input, not output. Speech is a fast, natural way to get information into a computer; listening to a machine talk back is not inspiring. The future of voice is therefore in replacing tedious tasks with speech, not in conversing with a computer. That means better form filling, richer search, and more efficient navigation in web and mobile applications.
For designers and developers, adding a voice modality to an existing graphical interface is a direct way to reduce friction — voice search can already be up to five times faster than a traditional search filtering UI, even on a desktop. The key is to add voice where a task can be accurately described in natural language, and to leave the conversational paradigm out of it.
