Using Multimodal AI Models For Your Applications (Part 3) — Smashing Magazine

Unified Architectures And What They Enable

Earlier in this series, the systems you built combined a dedicated vision-language model with a separate text-to-speech pipeline. That two-model approach works, but it means managing different runtimes, formats and output stages. The current generation of “any-to-any” models removes that friction by handling images or video, text, and audio in a single architecture. Reka and Gemini 1.5 Pro are the two we’ll look at here. Both can take spoken language or a still image as input and produce written or spoken output on their own, so you no longer need a second model for speech recognition or synthesis.

How Any-To-Any Models Differ From Their Predecessors

Multimodal systems have traditionally been assemblies of single-purpose components, but that has been changing with research systems that treat several modalities in one graph. Next-GPT and Apple’s 4M, for instance, demonstrate that a shared set of parameters can handle more than one input type without bolting on separate encoders and decoders.

Two notable reference points show different routes to that goal. DeepMind’s Gato uses a 1.2-billion-parameter decoder-only transformer with 24 layers, an embedding size of 2048 and a hidden size of 8196 in its feed-forward blocks, an architecture tuned for generalist behaviour. Its performance, though, still depends on extensive task-specific fine-tuning. GPT-4o condenses this further by training on several media types inside one model, which lets it switch between text, images and code without separate subsystems, improving generalization across tasks.

CoDi, from Microsoft, takes a multistage training approach aimed at keeping the number of tasks linear while opening up input-output combinations across modalities. Its design builds a shared multimodal space between the different encoders, then enables synchronized generation for related outputs — video with its corresponding audio is a good example.

Across these reference implementations, the technical scaffolding tends to be the same set of ideas:

• Shared representation space. Text becomes vectors, images become feature maps, audio becomes spectrograms or embeddings. With all inputs projected into one space, the model can handle them in a single forward pass instead of per‑modality branches.
• Attention mechanisms. Attention is what lets the network weigh the relevant parts of an input, whether that is a sentence, a region of an image, or a segment of speech.
• Cross-modal interaction. When text conditions the generation of an image, or an image conditions the interpretation of speech, that interaction usually happens in the shared representation space.
• Pre-training and fine-tuning. These systems are pre-trained across large mixed datasets in order to handle multiple modalities, then fine-tuned for final tasks.

Closing The Gap With A Two-Model Setup

Where earlier parts of this series paired LLaVA with Whisper for image understanding and spoken output, Reka and Gemini 1.5 Pro can now collapse that pipeline. Since both handle audio on the input side and the output side in addition to vision and text, one model covers what used to require separate speech recognition components — reducing integration overhead and simplifying the runtime. The trade-off this final part concentrates on is seeing how that change in architecture affects the way the application is assembled end to end.

Reka: A Multimodal Contender

Reka positions its models for developers who need multimodal understanding across images, video, audio, and long documents. The company also emphasizes code generation and execution as part of its offering. Their lineup includes three primary models:

• Reka Core: A 67-billion-parameter model aimed at complex reasoning and coding tasks.
• Reka Flash: A 21-billion-parameter model tuned for faster performance and flexibility.
• Reka Edge: A compact 7-billion-parameter model designed for on-device and low-latency use cases.

All three can be fine-tuned and deployed across cloud, on-premises, or on-device environments. Before writing any code, you can explore their capabilities via the playground, which supports image and video comprehension tests.

Reading Diagrams and Video

We started by uploading a diagram of the PaliGemma architecture to see how Reka Core handles a technical schematic. Asked for a detailed explanation, the model provided a breakdown that correctly identified the SigLIP image encoder, linear projection, concatenated tokens, and the Gemma processor. It accurately described how image and text inputs flow through the system to produce text output.

“This image is a schematic representation of a machine-learning model named PaliGemma... The image input is processed by a ‘SigLIP Image Encoder’... The output of this encoder is then subjected to a ‘Linear Projection’... Both the projected image data and the text input are then concatenated into ‘Concatenated Tokens’, which are then processed by a component labeled ‘Gemma’...”

The response was a clear, accurate interpretation of the architecture. Next, we tested video understanding. A short screen recording about the software Genny elicited a terse, less useful response: “This video is a screen recording of a group of friends discussing their plans for the weekend.” That fell short of what LLaVA produced for similar tasks in the previous part of this series.

Switching to Reka Flash, we used a longer video about the Parler TTS model. This time we framed the prompt around developer needs, asking how the tool could be used in projects. The model answered with a practical, organized set of suggestions:

• Improving accessibility for users with visual impairments or reading difficulties through spoken narration.
• Building more natural-sounding voice assistants and chatbots.
• Adding narration or instructions to e-books, education tools, and games.
• Generating spoken alerts and reminders when screens are not visible.
• Supporting language learning with correct pronunciation and intonation.
• Producing audiobooks from written content.
• Creating voiceovers for explainer videos.
• Providing audio feedback for actions, instructions, or performance reports.

This was a far stronger result than the earlier video test, offering genuinely actionable guidance for an AI developer.

Tackling PDFs with Reka Edge

To evaluate document understanding on the smallest model, we fed a PDF version of Meta AI’s “Responsible Use Guide” to Reka Edge. Prompted for how a developer could follow those guidelines, the model responded with a comprehensive list covering everything from familiarizing yourself with core principles to iterative development, ecosystem collaboration, and specific addendum considerations for Code Llama. The answers address monitoring, transparency, accountability, and staying current with research.

For this kind of dense, policy-oriented document, Reka Edge handled the extraction and synthesis well. Its response was detailed enough to serve as a practical checklist for responsible AI implementation.

Moving From Playground to Code

Once you are ready to move beyond the playground, you can sign up for a Reka account to obtain an API key. The Python SDK and a demo notebook are available, and the documentation lists which models are available for different tasks. That setup lets you see how these multimodal tools behave when integrated directly into your own applications.

Comparing Reka and Gemini: Deployment Patterns and Context Handling

When choosing between multimodal models for your application, the practical differences often come down to where the model runs and how much information it can process at once. Reka's design philosophy centers on flexibility for on-device deployment, while Gemini 1.5 Pro from Google DeepMind takes a different approach with its emphasis on extensive context handling in the cloud.

Reka's architecture makes it particularly attractive for scenarios requiring offline functionality or minimal latency. Applications that need to process inputs without a stable network connection, or those where every millisecond of response time matters, can benefit from running Reka models directly on edge devices.

Gemini 1.5 Pro, by contrast, leverages a Mixture-of-Experts (MoE) system that activates only the most relevant subnetworks for a given task. This approach enables complex reasoning without the computational overhead of a monolithic model. The key advantage, however, is the model's ability to process extremely long context windows, making it a strong candidate for analyzing large documents, extensive codebases, or multi-turn interactions that would overwhelm models with shorter context limits.

Hands-On Testing with Gemini 1.5

Google AI Studio provides immediate access to Gemini's capabilities, allowing for direct experimentation with image and video inputs. Testing begins with visual understanding tasks that probe the model's ability to interpret complex diagrams and extract structured information.

When presented with a flowchart depicting various Vision Language Model (VLM) tasks and asked to explain the specific details or concepts it represents, Gemini provides a comprehensive breakdown. The model identifies and describes four distinct task categories:

"The image is a flowchart illustrating different types of image understanding tasks with a Vision Language Model.

• Object Localization: This task aims to identify the precise location of objects within an image. The example question asks, 'Is one cat behind another?'
• Zero-shot Segmentation: This task involves segmenting an image into different regions based on a given label, even if the model has never seen that specific label before. The example question asks to segment 'striped cat.'
• Zero-shot Visual QA: This task requires answering questions about an image, even if the model has not been specifically trained to answer that type of question. The example question asks, 'What is the breed of these cats?'
• One-shot Learning with Instructions: This task involves providing the model with a single example and instruction to learn a new concept and apply it to other images. The example question asks, 'Striped cats are called tabby cats. What is the breed of the cats in the image?'

"

The model demonstrates a clear grasp of the diagram's semantic content, correctly categorizing each task type and understanding the distinctions between zero-shot and one-shot learning paradigms.

Food Recognition and Ingredient Identification

Moving from abstract diagrams to concrete visual content, the next test involves food recognition. When shown an Italian meal and prompted to list the dishes, their country of origin, and required ingredients, Gemini offers a reasonably detailed classification of the items.

"The image shows an Italian feast with the following dishes:

1. Seafood Stew: A rich and flavorful stew with seafood like clams, mussels, shrimp, fish, and vegetables in a tomato broth.
2. Spaghetti with Tomato Sauce: Classic spaghetti pasta tossed in a simple yet flavorful tomato sauce.
3. Meatballs in Tomato Sauce: Tender and juicy meatballs simmered in a rich tomato sauce.
4. Lasagna: Layered pasta with meat sauce, cheese, and tomato sauce.
5. Side Salad: A refreshing side salad with mixed greens, tomatoes, and other vegetables.
6. Garlic Bread: Toasted bread slices rubbed with garlic and olive oil.
7. Marinara Sauce: A simple tomato-based sauce.

This is a typical Italian meal with various dishes showcasing the flavors of Italian cuisine. As for the ingredients, you'll need tomatoes for many of these dishes, specifically for the sauces and the side salad."

The dish identification is accurate and the geographical attribution is correct, though the ingredient list remains incomplete. The model captures the central role of tomatoes in the meal but does not enumerate herbs, pasta, seafood, or the other components that would appear in a comprehensive shopping list.

Video Understanding: Action Recognition Meets Fine-Grained Analysis

Video comprehension is where multimodal models often diverge in capability. Using a screen-recording-style tutorial video, Gemini is asked what purpose the video serves and which key actions or processes it demonstrates.

"The video is a tutorial about the importance of friendship and how it can impact our lives. It demonstrates the use of a video editing software called 'Genmy' to create a video and add text subtitles. The video creator is using Genmy to:

• Record their voice: This is seen at the top of the screen with the mic icon.
• Add text to speech: You see the 'Text to Speech' option, which is likely being used to generate subtitles or captions for the video.
• Edit the timeline: They're working with a video timeline, adding and arranging different elements.
• Add voice-over: The speaker is adding commentary to the video.
• Create a visually appealing presentation: They're using a combination of video footage, text, and voice-over to create an engaging tutorial.

"

The response captures both the technical workflow and the elements visible in the user interface, correctly identifying the software and interpreting UI icons to infer actions like voice recording. This level of action recognition is relevant for developers building video analysis tools or automated review systems.

A second video test uses footage of the Parler TTS system, moving the task from action recognition to content summarization. Gemini is asked to summarize the video's contents and assess their relevance for software engineers.

"This video showcases different voice samples generated by a text-to-speech (TTS) system called 'Parler-TTS.' The video highlights various voice characteristics like gender, pace, pitch, and even the simulated environment. For software engineers, this video demonstrates the progress made in TTS technology. These are some insights:

• Voice Variety: The video shows that Parler-TTS can create diverse voices, potentially useful for applications needing different personas or tones.
• Fine-grained Control: The video emphasizes the ability to control voice parameters like pace and pitch, showcasing the potential for customizability.
• Quality and Naturalness: The video focuses on the quality of the generated speech, demonstrating the advancements made in speech synthesis, which is crucial for user experience.

"

The model identifies the TTS system by name, understands the significance of controlling voice attributes, and relates the demonstrations to developer-relevant concerns like speech quality and parameterization. These insights point toward practical implementation guidance for engineers integrating TTS capabilities into their applications.

Accessing the Model Programmatically

For developers ready to integrate Gemini 1.5 into their own applications, Google AI Studio includes a Get Code button that generates ready-to-use code snippets. The platform supports multiple programming languages, including Python, Swift, and Java, so you can pull the sample code directly into your development environment and adapt it for production use.