Generating Real-Time Audio Sentiment Analysis With AI — Smashing Magazine

Beyond Static Files: Instant Sentiment Scoring

Audio sentiment analysis is useful, but it becomes far more powerful when insights arrive in real time. Building on a previous tutorial that scored emotions in uploaded audio files, this article extends that tool with live transcription and multilingual sentiment scoring. The result: an app that records speech from your microphone, transcribes it on the fly, identifies the language being spoken, and assigns an emotional score to the words as they're spoken.

The finished product is available as a live demo on Hugging Face Spaces.

Two Core Technologies

The architecture relies on two complementary open-source tools. Whisper, OpenAI's automatic speech recognition model, handles the audio-to-text conversion alongside language identification. Gradio, a UI framework designed for machine learning interfaces, provides a user-friendly front end without requiring complex configuration or prior ML experience.

The previous iteration of this tool used the DistilBERT model for sentiment scoring. This version upgrades to roberta-base-go_emotions, a pre-trained model available on the Hugging Face Model Hub, which offers more granular emotional detection than simple positive/negative classification.

How Whisper Handles Speech Recognition

Automatic speech recognition (ASR) underpins everything from voice assistants to automated call routing. The technology has historically struggled with varied accents, background noise, and speech patterns. Whisper was trained on roughly 680,000 hours of multilingual, multitask supervised data gathered from the web, which helps it tackle these ASR pain points.

Whisper comes in multiple model sizes, with English-only variants (tiny.en, base.en, small.en, medium.en) available for single-language tasks.

The model uses a Seq2seq transformer encoder-decoder architecture. It ingests audio in 30-second segments and outputs corresponding text. For developers prioritizing English-only performance, the tiny.en and base.en variants deliver noticeably better results than their multilingual counterparts.

Setting Up the Environment

The entire application lives in a single app.py file. Beyond that, a requirements.txt file documents the project dependencies. All required libraries are installable with npm, though the language of the UI framework is Python:

!pip install gradio
!pip install transformers
!pip install git+https://github.com/openai/whisper.git

Once installed, the necessary modules are imported:

import gradio as gr
import whisper
from transformers import pipeline

This brings in Whisper for speech recognition, Gradio for the UI, and the pipeline function from Hugging Face Transformers to run the sentiment analysis model.

The initialization step loads Whisper for transcription and sets up the sentiment analyzer:

model = whisper.load_model("base")

sentiment_analysis = pipeline(
  "sentiment-analysis",
  framework="pt",
  model="SamLowe/roberta-base-go_emotions"
)

Wiring the Analysis Functions

The core logic relies on four Python functions, each responsible for a distinct step in the pipeline.

Function 1: analyze_sentiment(text)

This function accepts raw text and runs it through the pre-trained sentiment model. The output is a dictionary mapping detected sentiment labels — such as "optimistic" or "sad" — to their confidence scores.

def analyze_sentiment(text):
  results = sentiment_analysis(text)
  sentiment_results = {
    result[’label’]: result[’score’] for result in results
  }
return sentiment_results

Function 2: get_sentiment_emoji(sentiment)

A mapping translates each emotional label into a visual cue. An "optimistic" sentiment maps to a grinning face emoji, for instance. If a sentiment label has no direct mapping, the function returns an empty string.

def get_sentiment_emoji(sentiment):
  # Define the mapping of sentiments to emojis
  emoji_mapping = {
    "disappointment": "😞",
    "sadness": "😢",
    "annoyance": "😠",
    "neutral": "😐",
    "disapproval": "👎",
    "realization": "😮",
    "nervousness": "😬",
    "approval": "👍",
    "joy": "😄",
    "anger": "😡",
    "embarrassment": "😳",
    "caring": "🤗",
    "remorse": "😔",
    "disgust": "🤢",
    "grief": "😥",
    "confusion": "😕",
    "relief": "😌",
    "desire": "😍",
    "admiration": "😌",
    "optimism": "😊",
    "fear": "😨",
    "love": "❤️",
    "excitement": "🎉",
    "curiosity": "🤔",
    "amusement": "😄",
    "surprise": "😲",
    "gratitude": "🙏",
    "pride": "🦁"
  }
return emoji_mapping.get(sentiment, "")

Function 3: display_sentiment_results(sentiment_results, option)

Users can control the output format through a display option. There are two pathways: show the emoji alone, or pair the emoji with its confidence score. The function parses the sentiment and score results, applies the chosen format, and generates a formatted string for the UI.

def display_sentiment_results(sentiment_results, option):
sentiment_text = ""
for sentiment, score in sentiment_results.items():
  emoji = get_sentiment_emoji(sentiment)
  if option == "Sentiment Only":
    sentiment_text += f"{sentiment} {emoji}\n"
  elif option == "Sentiment + Score":
    sentiment_text += f"{sentiment} {emoji}: {score}\n"
return sentiment_text

Function 4: inference(audio, sentiment_option)

The main orchestration function takes the audio file and the display preference. It chains together the full inference process — language detection, transcription, and sentiment scoring — and returns all three data points for rendering in the Gradio interface.

def inference(audio, sentiment_option):
  audio = whisper.load_audio(audio)
  audio = whisper.pad_or_trim(audio)

  mel = whisper.log_mel_spectrogram(audio).to(model.device)

  _, probs = model.detect_language(mel)
  lang = max(probs, key=probs.get)

  options = whisper.DecodingOptions(fp16=False)
  result = whisper.decode(model, mel, options)

  sentiment_results = analyze_sentiment(result.text)
  sentiment_output = display_sentiment_results(sentiment_results, sentiment_option)

return lang.upper(), result.text, sentiment_output

With these functions in place, the front-end UI wires the live audio input to Whisper for transcription and feeds the resulting text into the sentiment pipeline, delivering real-time emotional insight as speech unfolds.

Building the Interface

With the transcription and sentiment functions ready, the remaining work is to assemble a Gradio layout that captures audio and renders the analysis results.

The steps below are specific to Gradio’s block-based UI framework; other frameworks will require different approaches.

Header Elements

The header carries a title, an image, and a short explanation of how sentiment scoring works. Storing these values in variables keeps the layout code compact:

title = """
🎤 Multilingual ASR 💬
"""
image_path = "/content/thumbnail.jpg"

description = """
  💻 This demo showcases a general-purpose speech recognition model called Whisper. It is trained on a large dataset of diverse audio and supports multilingual speech recognition and language identification tasks.


  📝 For more details, check out the \[GitHub repository\](https://github.com/openai/whisper).


  ⚙️ Components of the tool:


       - Real-time multilingual speech recognition

       - Language identification

       - Sentiment analysis of the transcriptions


  🎯 The sentiment analysis results are provided as a dictionary with different emotions and their corresponding scores.


  😃 The sentiment analysis results are displayed with emojis representing the corresponding sentiment.


  ✅ The higher the score for a specific emotion, the stronger the presence of that emotion in the transcribed text.


  ❓ Use the microphone for real-time speech recognition.


  ⚡️ The model will transcribe the audio and perform sentiment analysis on the transcribed text.

"""

Custom Styling

Gradio accepts custom CSS through a variable holding the style rules. This keeps visual adjustments separate from the component definitions:

custom_css = """
  #banner-image {
    display: block;
    margin-left: auto;
    margin-right: auto;
  }
  #chat-message {
    font-size: 14px;
    min-height: 300px;
  }
"""

Gradio Blocks

Gradio’s UI is organized around blocks, which are containers for layouts, components, and events. A block can attach the custom CSS defined above:

block = gr.Blocks(css=custom_css)

The header variables can then be placed inside the same block:

block = gr.Blocks(css=custom_css)

with block:
  gr.HTML(title)

with gr.Row():
  with gr.Column():
    gr.Image(image_path, elem_id="banner-image", show_label=False)
  with gr.Column():
    gr.HTML(description)

That renders the app title, image, description, and style rules together.

Form Component

The main form collects microphone audio, then delivers a transcript and a sentiment score in the format the user selects. Gradio provides a Group() container holding a Box() child — a pre-styled bordered element with rounded corners and padding:

with gr.Group():
  with gr.Box():

Within that Box(), we place the audio input, the radio buttons for analysis format, and the submit button:

with gr.Group():
  with gr.Box():
    # Audio Input
    audio = gr.Audio(
      label="Input Audio",
      show_label=False,
      source="microphone",
      type="filepath"
    )

    # Sentiment Option
    sentiment_option = gr.Radio(
      choices=["Sentiment Only", "Sentiment + Score"],
      label="Select an option",
      default="Sentiment Only"
    )

    # Transcribe Button
    btn = gr.Button("Transcribe")

Output Fields

Textbox() components serve as output areas for the predicted language, the transcription, and the sentiment result.

lang_str = gr.Textbox(label="Language")
text = gr.Textbox(label="Transcription")
sentiment_output = gr.Textbox(label="Sentiment Analysis Results", output=True)

Button Handler

The form’s Button() — labeled “Transcribe” — is wired to the inference() function defined earlier, mapping it to the expected inputs and outputs:

btn.click(
  inference,
  inputs=[
    audio,
    sentiment_option
  ],
  outputs=[
    lang_str,
    text,
    sentiment_output
  ]
)

Footer

The page footer credits OpenAI with a link to its GitHub repository.

gr.HTML(’’’
  <div class="footer">
    <p>Model by <a href="https://github.com/openai/whisper" style="text-decoration: underline;" target="_blank">OpenAI</a>
    </p>
  </div>
’’’)

Launching

Finally, the block is launched to render the interface:

block.launch()

Deployment on Hugging Face Spaces

The project already depends on Hugging Face’s Transformers library, so using its Spaces hosting platform for deployment is a natural fit. Spaces runs Python-based demos and experiments with minimal configuration.

Self-hosting is also possible, but Spaces integrates cleanly with the existing stack, making the deployment path straightforward.

Creating a Space

Start by creating a new Space, supplying the required metadata: a name (for example, “Real-Time-Multilingual-sentiment-analysis”), a license type such as BSD, the Gradio SDK, either free or paid hardware, and a visibility setting (public or private).

After creation, the Space can be cloned locally, or added as a Git remote to the existing repository.

Pushing Code

With app.py and requirements.txt ready, you can push them to the Space via standard Git commands from a terminal, or create and edit both files directly in the browser-based Space editor. Once the code is pushed, the Space shows a blue “Building” status while it compiles and starts the app.

Result

The completed application accepts an audio file, transcribes it to text, detects the language, classifies the emotion, and outputs a sentiment score.

The stack combines OpenAI’s Whisper for automatic speech recognition, four custom functions that drive the sentiment analysis, the pre-trained roberta-base-go_emotions model from the Hugging Face Hub, Gradio as the UI framework, and Hugging Face Spaces for deployment.