Integrating Image-To-Text And Text-To-Speech Models (Part 2) — Smashing Magazine

Building Conversational Vision Understanding

Building on the image-to-audio description tool from Part 1, this iteration takes a more ambitious approach: creating an application that can hold interactive, meaningful conversations about the visual content you provide. Instead of a one-way description, the tool becomes a conversational partner — you can upload an image or video and ask follow-up questions about what you see, much like chatting with a virtual assistant.

The first version of the app performed well but had limitations. Upload a photo of a dog, for example, and the resulting description might note “a dog sitting on a rock in front of a pool” without capturing the breed, the time of day, or the setting. The goal here is to move beyond surface-level descriptions and deliver richer, more complete insights through dialogue.

This approach is known as Conversational AI — technology that lets users talk to systems interactively about their input, whether that input is an image, a video, or another form of media.

From Visual Instruction Tuning to LLaVA

To enable this conversational capability, we turn to visual instruction tuning, a technique that helps large language models (LLMs) understand and follow instructions based on visual inputs. This method connects language and vision, allowing AI systems to respond to human queries that involve both text and images. Visual instruction tuning makes models capable of tasks like describing a scene in a photograph or answering questions about it.

This approach extends to specialized applications. LLaVAR, for instance, is a training method focused on handling PDFs, invoices, and text-heavy images — a useful area, though beyond the scope of this article's application.

The quality of such models depends heavily on the training data. Two notable datasets for visual instruction tuning stand out:

• Vision-CAIR: An English-language, multi-task dataset containing both human and machine-generated data. It provides high-quality, well-aligned image-text pairs created through conversations between two bots, as introduced in the MiniGPT-4 paper. This dataset yields more detailed image descriptions and works with predefined instruction templates for fine-tuning.
• LLaVA Visual Instruct 150K: GPT-generated multimodal instruction-following data built for visual instruction tuning. This dataset helps models aim for GPT-4 level vision and language comprehension.

LLaVA Architecture and Training

LLaVA — Large Language and Vision Assistant — is an open-source multimodal model from researchers at the University of Wisconsin, Microsoft Research, and Columbia University. Its open nature makes it easy to fine-tune and integrate. Critically, it understands and responds to complex visual information, even with unfamiliar images and instructions — exactly what we need for conversational analysis.

The architecture cleverly reuses existing models rather than reinventing the wheel:

• CLIP VIT-L/14, OpenAI's advanced vision model that learns visual concepts from natural language descriptions and can handle visual classification tasks using zero-shot capabilities.
• Vicuna, an open-source chatbot created by fine-tuning LLaMA on 70,000 user-shared conversations. It performs remarkably well despite its modest training cost of roughly $300, even compared to alternatives like Alpaca.

Training LLaVA proceeds in two key stages:

1. Pre-training for Feature Alignment: The model syncs visual and language features by updating a projection matrix — a bridge between the CLIP visual encoder and the Vicuna language model. Using a subset of the CC3M dataset, this step maps images and text into the same feature space.
2. End-to-End Fine-Tuning: The entire model is fine-tuned, with the visual encoder's weights frozen while the projection layer and language model adapt. This stage targets specific use cases:
   • Instruction-based fine-tuning for general applications with datasets built around following visual-and-textual instructions;
   • Scientific reasoning fine-tuning for specialized domains that demand complex reasoning on detailed questions.

This smart combination of state-of-the-art visual and language components makes LLaVA highly effective for applications needing both visual and conversational AI.

Whisper Large-v3 for Speech

Turning text-based responses into natural, audible answers calls for Whisper, OpenAI's speech recognition and translation model. In this version of the application, we use the newer large-v3 release, which offers better performance and speed than its predecessors.

Whisper large-v3 introduces two notable improvements over earlier versions:

• Better inputs: It uses 128 Mel frequency bins instead of 80. Mel frequency bins break audio into segments the model can process; more bins capture finer detail for better understanding.
• More training data: It was trained on 1 million hours of weakly labeled audio plus 4 million hours of pseudo-labeled audio collected from Whisper large-v2, running for 2.0 epochs over the mix.

Whisper models come in several sizes, from tiny to large, each offering different trade-offs between performance and resource usage for the same core capability.

Building the Assistant: LLaVA and Whisper

For the image and video input side, we’ll use LLaVA. The app will accept both images and videos, adding a layer of versatility. The speech output feature remains, letting users hear the assistant’s responses. Whisper handles the audio transcription. We’ll continue with the Gradio framework for the UI, though other models and frameworks can be swapped in as needed.

Setup and Dependencies

First, install the required packages. This includes the transformers library for loading LLaVA and Whisper, bitsandbytes for quantization, gtts for text-to-speech, and moviepy for video frame extraction.

#python
!pip install -q -U transformers==4.37.2
!pip install -q bitsandbytes==0.41.3 accelerate==0.25.0
!pip install -q git+https://github.com/openai/whisper.git
!pip install -q gradio
!pip install -q gTTS
!pip install -q moviepy

Once installed, import these libraries into the environment. The following code uses colab for this step:

#python
import torch
from transformers import BitsAndBytesConfig, pipeline
import whisper
import gradio as gr
from gtts import gTTS
from PIL import Image
import re
import os
import datetime
import locale
import numpy as np
import nltk
import moviepy.editor as mp

nltk.download('punkt')
from nltk import sent_tokenize

# Set up locale
os.environ["LANG"] = "en_US.UTF-8"
os.environ["LC_ALL"] = "en_US.UTF-8"
locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')

Model Configuration and Quantization

Set up a 4-bit quantization configuration to make the LLaVA model more efficient in memory usage and performance.

#python

# Configuration for quantization
quantization_config = BitsAndBytesConfig(
  load_in_4bit=True,
  bnb_4bit_compute_dtype=torch.float16
)

# Load the image-to-text model
model_id = "llava-hf/llava-1.5-7b-hf"
pipe = pipeline("image-to-text",
  model=model_id,
  model_kwargs={"quantization_config": quantization_config})

# Load the whisper model
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
model = whisper.load_model("large-v3", device=DEVICE)

This code applies a four-bit quantization setting to reduce memory footprint. We then load the LLaVA model with these settings and load the Whisper model, selecting the device based on GPU availability.

Note: We are using llava-v1.5-7b as the model. Other LLaVA versions can be explored. For Whisper, the “large” size is used, but you can switch to “medium” or “small” for experiments.

To run the assistant, we implement five core functions:

1. Handling conversation history.
2. Converting images to text.
3. Converting videos to text.
4. Transcribing audio.
5. Converting text to speech.

After defining these individually, we create a main function to tie them together. The code for each is in the following sections.

Conversation History

Start with the conversation history and a function that logs it:

#python

# Initialize conversation history
conversation_history = []

def writehistory(text):
  """Write history to a log file."""
  tstamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
  logfile = f'{tstamp}_log.txt'
  with open(logfile, 'a', encoding='utf-8') as f:
    f.write(text + '\n')

Image to Text

Create a function to convert images to text with LLaVA using iterative prompts:

#python
def img2txt(input_text, input_image):
  """Convert image to text using iterative prompts."""
  try:
    image = Image.open(input_image)
      
    if isinstance(input_text, tuple):
      input_text = input_text[0]  # Take the first element if it's a tuple

      writehistory(f"Input text: {input_text}")
      prompt = "USER: <image>\n" + input_text + "\nASSISTANT:"
      while True:
        outputs = pipe(image, prompt=prompt, generate_kwargs={"max_new_tokens": 200})

          if outputs and outputs[0]["generated_text"]:
            match = re.search(r'ASSISTANT:\s*(.*)', outputs[0]["generated_text"])
            reply = match.group(1) if match else "No response found."
            conversation_history.append(("User", input_text))
            conversation_history.append(("Assistant", reply))
            prompt = "USER: " + reply + "\nASSISTANT:"
            return reply  # Only return the first response for now
          else:
            return "No response generated."
  except Exception as e:
    return str(e)

Video to Text

For video inputs, extract frames and analyze them to produce a text description:

#python
def vid2txt(input_text, input_video):
  """Convert video to text by extracting frames and analyzing."""
  try:
    video = mp.VideoFileClip(input_video)
    frame = video.get_frame(1)  # Get a frame from the video at the 1-second mark
    image_path = "temp_frame.jpg"
    mp.ImageClip(frame).save_frame(image_path)
    return img2txt(input_text, image_path)
  except Exception as e:
    return str(e)

Audio Transcription

Add a function that uses Whisper to transcribe audio prompts:

#python
def transcribe(audio_path):
  """Transcribe audio to text using Whisper model."""
  if not audio_path:
    return ''

  audio = whisper.load_audio(audio_path)
  audio = whisper.pad_or_trim(audio)
  mel = whisper.log_mel_spectrogram(audio).to(model.device)
  options = whisper.DecodingOptions()
  result = whisper.decode(model, mel, options)
  return result.text

Text to Speech

Finally, define a function that converts the text response into speech:

#python
def text_to_speech(text, file_path):
  """Convert text to speech and save to file."""
  language = 'en'
  audioobj = gTTS(text=text, lang=language, slow=False)
  audioobj.save(file_path)
  return file_path

The main function that orchestrates all these pieces looks like this:

#python

def chatbot_interface(audio_path, image_path, video_path, user_message):
  """Process user inputs and generate chatbot response."""
  global conversation_history

  # Handle audio input
  if audio_path:
    speech_to_text_output = transcribe(audio_path)
  else:
    speech_to_text_output = ""

  # Determine the input message
  input_message = user_message if user_message else speech_to_text_output

  # Ensure input_message is a string
  if isinstance(input_message, tuple):
    input_message = input_message[0]

  # Handle image or video input
  if image_path:
    chatgpt_output = img2txt(input_message, image_path)
  elif video_path:
      chatgpt_output = vid2txt(input_message, video_path)
  else:
    chatgpt_output = "No image or video provided."

  # Add to conversation history
  conversation_history.append(("User", input_message))
  conversation_history.append(("Assistant", chatgpt_output))

  # Generate audio response
  processed_audio_path = text_to_speech(chatgpt_output, "Temp3.mp3")

  return conversation_history, processed_audio_path

Building the UI with Gradio

The interface uses Gradio for quick prototyping. Users can record or upload audio prompts, type questions, upload videos, and see the conversation history.

#python

# Define Gradio interface
iface = gr.Interface(
  fn=chatbot_interface,
  inputs=[
    gr.Audio(type="filepath", label="Record your message"),
    gr.Image(type="filepath", label="Upload an image"),
    gr.Video(label="Upload a video"),
    gr.Textbox(lines=2, placeholder="Type your message here...", label="User message (if no audio)")
  ],
  outputs=[
    gr.Chatbot(label="Conversation"),
    gr.Audio(label="Assistant's Voice Reply")
  ],
  title="Interactive Visual and Voice Assistant",
  description="Upload an image or video, record or type your question, and get detailed responses."
)

# Launch the Gradio app
iface.launch(debug=True)

Here’s a preview of the resulting app interface:

Beyond LLaVA: Any-to-Any Models

LLaVA handles vision and text well but requires a separate automatic speech recognition (ASR) model to cover audio. Multimodal “any-to-any” models can process and integrate multiple modalities without that extra step. They handle image-to-text, video-to-text, text-to-speech, speech-to-text, and more, simplifying both architecture and workflow.

Examples of Multimodal Models

Several models can integrate images, text, audio, and other data types. These are worth exploring for future projects.

CoDi

CoDi (Composable Diffusion) is versatile and can take text, images, audio, and video as input, producing different media outputs. Developed by researchers from the University of North Carolina and Microsoft Azure, it uses Composable Diffusion to sync data types, such as aligning audio and video, and can generate outputs not present in the original training data.

ImageBind

ImageBind from Meta binds together data from six modalities: images, video, audio, text, depth, and thermal data. Without explicit supervision, it learns how these data types relate, making it useful for systems that combine multiple data types — for example, pairing 3D sensor data with IMU data for virtual world design or richer media search.

Gato

Gato is a generalist agent that handles a wide range of tasks with the same network — from games and chat to image captioning and robot arm control. Its key trait is switching between task types and outputs without model changes.

GPT-4o

GPT-4o from OpenAI is a multimodal LLM handling text, audio, image, and video inputs and producing text, audio, and image outputs. Audio responses come in 232ms to 320ms, close to real-time conversation. A smaller GPT-4o Mini version also exists, showing that smaller models can perform well against larger ones, a growing trend.

Summary

From setting up LLaVA for image and video understanding to integrating Whisper large-v3 for speech recognition, we have covered a complete workflow for a multimodal assistant. Exploring models like CoDi and GPT-4o demonstrates the range of options available for handling diverse data types in one system.