How To Enable Collaboration In A Multiparty Setting — Smashing Magazine

A Third Party at the Table: Designing for Multiparty Advisory Sessions

Banking consultations are a demanding blend of conversation, data entry, and client education. While the consultant manages a high cognitive load, the client frequently faces a state of passivity rooted in information asymmetry — the consultant simply has more and better information. To address these challenges in trust, transparency, and overload, a research project introduced a digital agent named “Mo.” Operating on an interactive tabletop projection, Mo acts as a third interaction partner throughout the consultation, supporting everything from note-taking to portfolio composition.

Rather than replacing the consultant, the goal is to free them to focus on the client’s needs. The system fosters a collaborative exchange where all information is equally visible and accessible. Content can be placed anywhere on the table through natural, haptic interactions, and Mo participates using Swiss-German dialects to remain approachable. Several key design elements make this multiparty collaboration possible:

• Shared Device. A single, interactive surface makes all information equally visible and interactable for every participant.
• Collaborative Digital Agent. The agent uses human modes of communication, social cues, and local dialects to become more accessible and accepted by users.
• Comprehensible User Interfaces. Multimodal communication leverages different output channels to convey information at varying levels of complexity.
• Speech Patterns for Voice User Interfaces. Direct commands appear unnatural in group settings; varied speech and turn-taking patterns help the agent integrate naturally into the conversation.

Designing Voice Interactions for Two-Party Conversation

Voice as an input method eliminates the need to shift context to a keyboard or mouse. During a consultation, this allows the advisor to maintain a personal connection with the client while capturing or retrieving data. More importantly, voice excels at handling complex information entry — purchasing stocks, for example, requires multiple parameters such as title and share count. In a GUI, each of these must be entered laboriously; a VUI can capture the entire transaction in a single sentence.

Despite that strength, users treat voice interfaces with significant skepticism. Designing intuitive interactions requires understanding fundamental speech patterns and their trade-offs in a multiparty context.

Command and Control

Popularized by assistants like Siri, Alexa, and Google Assistant, this pattern relies on direct commands often preceded by a “wake word.” Users get control and predictability, but the pattern introduces unnatural interruptions and pauses into a fluid conversation.

Conversational

In this pattern, the agent extracts intents directly from the context of natural speech, without a preceding command, and may even initiate dialogue itself. It is less common in production systems but offers a more effortless interaction. However, during an advisory meeting this proved unpredictable: when a consultant and client actively discuss stock titles, not every utterance signals a transaction. Distinguishing between an option being debated and an actual purchase intent is nearly impossible for the agent, which led users to perceive spontaneous transaction detection as surprising and even irritating.

Conversational Confirmation

To reconcile natural interactions with predictable behavior, a new pattern was introduced. Typical transaction dialogues naturally follow a structure of intent, confirmation, and reconfirmation:

Interlocutor 1: We then buy 20 shares of Smashing Media Stocks (intent).
Interlocutor 2: Yes, let’s do that (confirmation).
Interlocutor 1: All right then, let’s buy Smashing Media Stocks (reconfirmation).

In the standard Conversational Pattern, the system executes a transaction after the first utterance. In the Conversational Confirmation pattern, the agent waits for both parties to confirm before executing after the third utterance. By aligning with the natural rules of human conversation, the approach meets user expectations and removes the perception of unpredictability.

The comparison yields a clear set of recommendations for choosing a speech pattern:

1. The Command and Control Pattern gives users a greater sense of control and security.
2. Command and Control should serve as a fallback when the agent fails to understand an intent.
3. The Conversational Pattern is appropriate for passive information gathering, such as logging.
4. For collaborative counseling sessions, the Conversational Confirmation Pattern greatly enhances the experience and to a more natural multiparty conversation.

The Shared Device: Moving Beyond Single-Player

Personal computers, phones, and tablets have conditioned us to interact with technology in “single-player mode.” However, in a collaborative advisory session, this creates exclusion: when an advisor displays stock trends on a personal screen, at least one participant is blocked from full visibility and access, deepening the information asymmetry.

Replacing that private screen with an interactive tabletop projection changes the dynamic fundamentally. The physical space itself becomes the interface — content is no longer bound to a display but can be moved and placed freely in the interaction space using haptic tokens. These tokens are equally accessible to both parties, encouraging novice users to engage directly with the surface.

The shared device also improves comprehension. As the agent updates portfolio visualizations in real time, the impact of a proposed transaction can be directly grasped — and physically pulled closer for discussion. The result is a transparent approach to information that increases the user’s understanding of bank-specific processes. That transparency and mutual visibility improves trust in the advisory service and promotes more interaction between customer and advisor.

Multimodal systems achieve collaboration because they adhere to several critical principles:

1. Information is equally accessible to all parties, reducing information asymmetry, fostering a shared understanding, and building trust.
2. A single user interface can be operated collectively by multiple interaction partners, increasing engagement.
3. Human multisensory communication is transferred naturally to the digital space, improving ease of use.
4. Multimodal output channels allow information to be better comprehended.

Making Room For A Third Partner

In a consultation setting, the goal is to design an intelligent agent that actively reduces the consultant’s cognitive load. For that agent to be effective, it must be perceived as a trustworthy and even likable collaboration partner. This requires moving beyond simple command execution and incorporating the social dynamics that make human teamwork work.

Why Social Cues Matter

Social cues are the verbal and non-verbal signals — eye contact, facial expressions, tone of voice, body language — that guide interactions by shaping our perceptions and reactions. They carry social and contextual information that is essential for mutual understanding. For a digital agent to be approachable and credible, those same cues need to be translated across different modalities.

On the visual side, the agent's embodiment allows for facial expressions and body language that mirror human communication. These features can also be used to highlight important system context or indicate status. In the example from the banking consultation prototype, the agent uses these visual channels to become a more expressive and understandable presence.

Voice interaction follows a similar logic. A hallmark of natural conversation is confirming an action with a short utterance like “mhm” or “ok.” Applying that pattern to the agent’s voice feedback makes the system feel more transparent and human.

One critical insight is that the speech pattern used by the agent heavily shapes user perception. If the agent is treated merely as a recipient of direct commands, it is relegated to a subordinate role. Conversely, when the system independently recognizes conversational intent, users perceive it as more intelligent and trustworthy — a peer rather than a servant.

Mo And The Value Of Transparency

Even with advances in Swiss German speech recognition, misrecognized transactions occur. Rather than hiding these flaws, the design leverages the agent to make system processes more understandable. This adheres to a classic usability heuristic: the more comprehensible the system, the more users feel empowered to interact with it, and the more they trust it.

In the portfolio adjustment phase, transactions are added and removed through the agent. When confidence in recognition is low, the agent “Mo” checks in to confirm whether the transaction was understood correctly.

This behavior follows ordinary conversational norms: an uncertain interlocutor speaks up, apologizes, and asks for confirmation. If the transaction was indeed misunderstood, the user can correct it via touch and a Microsoft Dial scrolling token rather than repeating the command. These alternative inputs were a deliberate choice to avoid cascading voice errors and user frustration. By giving the user direct control during an error, the system builds a stronger sense of agency, acceptance, and readiness to collaborate.

Key takeaways:

• Social cues make an agent more approachable and trustworthy. They are a vital instrument for conveying context and enabling effective system feedback.
• Involving the agent in explaining system processes (Explainable AI) improves acceptance and trust in both the agent and the underlying system.

The Future Of Advisory Sessions

Across legal, healthcare, insurance, and banking, the quality of counseling depends on two factors. The first is the advisor's ability to give the client undivided attention. The second is structuring the session to ensure equal access to information for all participants, presented in a way that is clear even to those without expertise. Clients who feel informed and self-determined are more likely to become loyal customers.

A shared interaction space offers a direct remedy to information asymmetry. In a physical meeting, an interactive tabletop projection creates a democratic environment where every participant has equivalent access to data — a result that is unattainable with personal devices.

Digital and remote consultations shift the interaction space but not the fundamental need for transparency and collaboration. The medium changes, yet the requirements evolve accordingly. Even in standard mobile or desktop interfaces, intentionally using multiple modalities improves comprehensibility.

To genuinely reduce the consultant’s workload, the system should manage time-consuming, background tasks. While voice interaction is still unfamiliar to many users and can fall short of expectations, speech processing is advancing. The time to consider what natural, collaborative interactions with assistants could look like is now — before the technology has fully caught up.