Using AI To Detect Sentiment In Audio Files — Smashing Magazine

Capturing Tone From Spoken Words

Written communication tools like Grammarly have made it common to check the tone of text before hitting send. The same idea can be extended to recorded speech: an application that ingests an audio file, converts the speech to text, and then evaluates that text for positive or negative sentiment. The workflow combines speech-to-text, natural language processing, and a lightweight UI layer.

Such a tool has obvious value for podcasters reviewing their own episodes, customer service teams monitoring support calls, and product teams collecting spoken feedback. But the underlying components are general enough that audio sentiment analysis can apply in healthcare, market research, and many other fields where understanding the emotional content of a conversation matters.

The implementation is built on three straightforward steps:

• Upload an audio file to the interface.
• Transcribe the spoken content into text.
• Compute a sentiment score for the resulting transcript.

Three technologies handle these tasks: Hugging Face Transformers for machine learning and NLP, a speech recognition model, and Streamlit for the web application.

Why Sentiment Analysis Matters Beyond Text

Text-only analysis misses the tonal cues present in natural speech. An audio sentiment analyzer closes that gap in a number of concrete scenarios:

• Call centers. Agents can assess customer mood in real time and tailor their responses for better empathy and personalization.
• Voice assistants. Developers of voice-based interfaces can improve response accuracy by understanding speakers’ emotional states and refining the underlying NLP models.
• Surveys. Teams collecting spoken survey responses can spot satisfaction trends and identify areas for improvement without manually listening to every recording.

Healthcare is another strong use case. Providers could analyze patient feedback conversations to discover pain points in interactions and improve care quality. Market researchers could process audio from interviews or focus groups to measure audience reactions to branding and products. Product teams might even replace written stakeholder feedback with recorded verbal commentary and run that through a sentiment model during design reviews.

Tools For The Job

Natural Language Processing With Transformers

The Hugging Face Transformers library acts as the backbone of the sentiment analysis. It exposes a large collection of pre-trained neural network models through a single API. More importantly for this project, the library covers speech recognition and audio classification tasks in addition to supporting major NLP workflows like text classification, named entity recognition, question answering, summarization, translation, and generation.

This means the audio analyzer does not train a model from scratch. Instead, Transformers supplies pre-trained checkpoints that the application calls from its Python code.

A UI Framework With Streamlit

Streamlit provides predefined components for quickly assembling interactive data applications. It fits well with building a command-line-friendly audio analysis tool because the integration requires minimal boilerplate, and test deployments are available directly through the framework's native service.

Putting the Pieces Together

With the core technical concepts defined, we can now walk through the implementation. The goal is a Streamlit application that accepts an audio file upload, transcribes the speech, and returns a sentiment score. The main steps are:

1. Setting up the environment and file structure.
2. Building the user interface with Streamlit.
3. Defining the sentiment analysis logic using the Hugging Face Transformers library.
4. Integrating speech recognition for transcription.
5. Connecting the UI to the analysis functions.

Project Structure and Dependencies

The application is built in a single, simple directory containing three files:

• app.py: The main script for the Streamlit application.
• requirements.txt: Specifies the project's Python dependencies.
• README.md: Documentation for the project.

The initial code imports the necessary libraries: os for system operations, traceback for error handling, streamlit (st) for the UI, speech_recognition (sr) for audio transcription, and pipeline from Transformers for the sentiment analysis model.

import os
import traceback
import streamlit as st
import speech_recognition as sr
from transformers import pipeline

Designing the User Interface

Streamlit provides a straightforward path to a clean UI. First, we configure the layout to use the wide format, giving us more horizontal space for results and controls.

st.set_page_config(layout="wide")

The main page then includes a title and descriptive text. A sidebar holds the app description and the file upload control, while the main area is reserved for displaying the transcription and sentiment score.

// app.py
st.title("🎧 Audio Analysis 📝")
st.write("[Joas](https://huggingface.co/Pontonkid)")

The sidebar is created with Streamlit's st.sidebar functionality. Within it, we add a file uploader configured to accept WAV audio files, along with an "Upload" button to trigger the processing workflow. Restricting the file type to WAV is a design choice; the uploader can be configured to accept other formats as needed.

// app.py
st.sidebar.title("Audio Analysis")
st.sidebar.write("The Audio Analysis app is a powerful tool that allows you to analyze audio files and gain valuable insights from them. It combines speech recognition and sentiment analysis techniques to transcribe the audio and determine the sentiment expressed within it.")

// app.py
st.sidebar.header("Upload Audio")
audio_file = st.sidebar.file_uploader("Browse", type=["wav"])
upload_button = st.sidebar.button("Upload")

Implementing the Sentiment Analysis Function

The sentiment analysis process follows a specific plan: load a pre-trained NLP model, use it to analyze the transcribed text, and return the score with its label.

The function leverages a pre-trained model for text classification hosted on the Hugging Face hub. The chosen model is DistilBERT, which is lightweight and well-suited for this classification task.

// app.py
def perform_sentiment_analysis(text):
  model_name = "distilbert-base-uncased-finetuned-sst-2-english"

The analysis is executed through the Transformers pipeline() function. This function returns a result from which we assign and return variables for the sentiment label and score.

// app.py
def perform_sentiment_analysis(text):
  model_name = "distilbert-base-uncased-finetuned-sst-2-english"
  sentiment_analysis = pipeline("sentiment-analysis", model=model_name)

// app.py
def perform_sentiment_analysis(text):
  model_name = "distilbert-base-uncased-finetuned-sst-2-english"
  sentiment_analysis = pipeline("sentiment-analysis", model=model_name)
  results = sentiment_analysis(text)

// app.py
def perform_sentiment_analysis(text):
  model_name = "distilbert-base-uncased-finetuned-sst-2-english"
  sentiment_analysis = pipeline("sentiment-analysis", model=model_name)
  results = sentiment_analysis(text)
  sentiment_label = results[0]['label']
  sentiment_score = results[0]['score']
  return sentiment_label, sentiment_score

This defines the complete perform_sentiment_analysis() function used by the app.

Adding Speech Recognition

The transcription is handled by a transcribe_audio() function that relies on the speech_recognition library. The process involves initializing a recognizer object, opening the uploaded audio file with the library's AudioFile function, and recording the audio data. The text is then extracted using the Google Speech Recognition API via the recognize_google() method.

// app.py
def transcribe_audio(audio_file):
  r = sr.Recognizer()
  with sr.AudioFile(audio_file) as source:
    audio = r.record(source)
    transcribed_text = r.recognize_google(audio)
  return transcribed_text

The main application logic is contained within a main() function. It first checks for two conditions: an audio file has been uploaded, and the upload button has been clicked. Only when both are true does it proceed with the transcription and sentiment analysis.

// app.py
def main():
  if audio_file and upload_button:
    try:
      transcribed_text = transcribe_audio(audio_file)
      sentiment_label, sentiment_score = perform_sentiment_analysis(transcribed_text)

Displaying the Results and Handling Errors

With the core logic in place, the next step is to hook it up to the UI. This involves setting up two main header elements, a text area for the transcription, and conditional logic with emoji icons to visually represent the sentiment outcome. If a sentiment label is empty, Streamlit's st.empty() is used to leave the section blank.

// app.py
st.header("Transcribed Text")
st.text_area("Transcribed Text", transcribed_text, height=200)
st.header("Sentiment Analysis")
negative_icon = "👎"
neutral_icon = "😐"
positive_icon = "👍"

// app.py
if sentiment_label == "NEGATIVE":
  st.write(f"{negative_icon} Negative (Score: {sentiment_score})", unsafe_allow_html=True)
else:
  st.empty()

if sentiment_label == "NEUTRAL":
  st.write(f"{neutral_icon} Neutral (Score: {sentiment_score})", unsafe_allow_html=True)
else:
  st.empty()

if sentiment_label == "POSITIVE":
  st.write(f"{positive_icon} Positive (Score: {sentiment_score})", unsafe_allow_html=True)
else:
  st.empty()

To help users interpret the score, we use Streamlit's st.info() element to display an informational message explaining the sentiment score results.

// app.py
st.info(
  "The sentiment score measures how strongly positive, negative, or neutral the feelings or opinions are."
  "A higher score indicates a positive sentiment, while a lower score indicates a negative sentiment."
)

An except block is included for robustness. If any exception occurs during processing, an error message is shown to the user with st.error(), and the full exception traceback is printed for debugging via traceback.print_exc().

// app.py
except Exception as ex:
  st.error("Error occurred during audio transcription and sentiment analysis.")
  st.error(str(ex))
  traceback.print_exc()

The final code block ensures the main() function is executed when the script is run as the main program. This standard Python pattern prevents the app from launching when the script is imported as a module.

// app.py
if __name__ == "__main__": main()

Deployment and Hosting

The deployment phase is handled through the Streamlit Community Cloud platform, which offers free and straightforward hosting. The process assumes accounts exist for both GitHub and Streamlit Community Cloud. GitHub serves as the code repository that Streamlit connects to, allowing it to fetch and deploy the application files.

There are three primary steps to get the app live:

1. Create a GitHub repository to store the code for management and collaboration.
2. Create the Streamlit application on the Community Cloud, linking it to the GitHub repository.
3. Configure deployment settings on the cloud platform, including the Python version and any required environment variables.

Once these steps are complete, Streamlit's automation handles the build and deployment process every time changes are pushed to the main branch of the connected GitHub repository.

Final Thoughts

This workflow demonstrates how an app can accept an audio file, transcribe the spoken content, analyze the resulting text, and return a sentiment score indicating whether the speaker's tone is positive or negative. The entire build relies on just two main technological pillars: the Hugging Face Transformers library for the language model and the Streamlit framework for the user interface, which conveniently includes deployment and hosting capabilities. This combination is sufficient to pull everything together into a functional tool.