From waveforms to frequency bins

A Fourier transform converts a signal between the time domain and the frequency domain. The idea becomes concrete once you manipulate a sine wave's parameters: amplitude scales the wave vertically and controls loudness, frequency changes pitch, and phase shifts the wave horizontally — a factor that only matters when mixing multiple waves. In stereo, left and right channels can interfere, even cancelling each other out entirely.

Sine waves aren't the only useful waveform. Adding enough cosine waves can approximate a square wave: one cosine sounds soft, but as you stack harmonics, the sum increasingly resembles a square wave and sounds sharper and louder. The Fourier transform is the operation that extracts the amplitude and phase of the cosine components needed to reconstruct any input signal, including real-world recordings.

Chunking and the Hann window

A fast Fourier transform (FFT) requires choosing an input size, typically a power of two. With a size of 1024, processing an entire song means splitting the input into chunks. That reconstruction sounds wrong, though — discontinuities appear at chunk boundaries because the analysis implicitly assumes each chunk is cyclical. The solution is a Hann windowing function applied to overlapping chunks.

The Hann window is designed so that two adjacent, half-shifted windows sum to a constant 1. By sampling with Hann windows and reconstructing via overlap-add, we avoid the spectral leakage at chunk boundaries. Reconstruction then becomes correct, except at the start and end where only a single window exists, producing artifacts. Real-world codecs like MP3 solve this with silence padding, which is why gapless playback took time to standardize — encoders had to start recording how many samples to skip at each end in metadata headers.

The Gabor limit

A spectrogram forces a tradeoff between time and frequency resolution. FFT computation is O(N log N), so size isn't strictly a computational constraint — but a larger FFT reduces time resolution. This reflects the Heisenberg-Gabor limit: you cannot sharply localize a signal in both the time and frequency domains simultaneously.

In a spectrogram, both matter since we plot frequency over time. The compromise means choosing between blurriness on the horizontal axis or the vertical axis. A window size of 4096 with a 128-sample offset creates 97% overlap between windows, which causes some smearing but produces smooth horizontal motion. Zero overlap with the same FFT size yields a compressed, slow-moving visualization; shrinking the FFT to compensate sacrifices significant frequency resolution.

Interpolation and color

Interpolation masks the frequency-bin boundaries. The computed bin index is a floating-point value: when drawing a pixel at bin index 3.5, we mix bins 3 and 4 equally. Disabling interpolation reveals hard edges between frequency bins, reducing visual quality.

Color mapping is another key design choice. A simple black-white gradient from 0 to 1 is hard to read — following a melody becomes much easier with a perceptually uniform color map. The "colorcet" collection provides such maps, and a Rust crate exists for color gradients generally, making it straightforward to apply these perceptually tuned palettes instead of relying on trial-and-error.

Frequency Scaling the Human Way

That brings us to the vertical axis. A naive linear plot from 20 Hz to 20 kHz looks tidy—harmonics are evenly spaced—but it doesn't reflect how we perceive pitch. Worse, it devotes a huge chunk of the display to the relatively uneventful 10 kHz–20 kHz band while cramming together the lower frequencies where all the real action happens.

Our fix is two-pronged. First, switch to a logarithmic scale:

fn get_frequency_breakpoints(max_freq: f32) -> [(f32, f32); 8] { [ (0.0, 20.0), // 0% -> 20 Hz (0.05, 100.0), // 5% -> 100 Hz (0.15, 500.0), // 15% -> 500 Hz (0.3, 1000.0), // 30% -> 1 kHz (0.45, 2000.0), // 45% -> 2 kHz (0.7, 5000.0), // 70% -> 5 kHz (0.9, 10000.0), // 90% -> 10 kHz (1.0, max_freq), // 100% -> just below Nyquist ] }

Then apply log interpolation between those bounds:

// Non-linear frequency mapping functions using piecewise scaling fn normalized_to_frequency(normalized_y: f32, sample_rate: f32) -> f32 { let max_freq = sample_rate * 0.499; let breakpoints = get_frequency_breakpoints(max_freq); // Find which segment we're in for i in 0..breakpoints.len() - 1 { let (y1, f1) = breakpoints[i]; let (y2, f2) = breakpoints[i + 1]; if normalized_y >= y1 && normalized_y <= y2 { // Linear interpolation within this segment let t = (normalized_y - y1) / (y2 - y1); // Use log interpolation for frequency let log_f1 = f1.ln(); let log_f2 = f2.ln(); let log_freq = log_f1 + t * (log_f2 - log_f1); return log_freq.exp(); } } breakpoints[breakpoints.len() - 1].1 }

The result is a much better compromise between detail and readability. Building a spectrogram turned out to involve far more design choices than expected—like any tool, it has to be shaped around the person using it.

Capturing System Audio

Now for some engineering details. My earlier audio project in The Science of Loudness read files from disk. This time I wanted to visualize whatever sound is playing on the machine—music apps, browsers, anything. RogueAmoeba's Loopback makes this trivial by presenting a virtual output device that also appears as an input, just like a microphone:

List of macOS input devices showing MacBook Pro Microphone, dusk Microphone (my phone), and Loopback Audio.

So we "just" capture from that virtual input. The cpal crate handles the plumbing: pick an input device, create a stream with a callback:

device.build_input_stream( &config.config(), move |data: &[f32], _: &_| { // do something with data }, err_fn, None, )?

That's for 32-bit floating samples; supporting other formats means matching on the audio config:

let stream = match config.sample_format() { cpal::SampleFormat::F32 => { device.build_input_stream( &config.config(), move |data: &[f32], _: &_| { // do something with f32 samples }, err_fn, None, )? } cpal::SampleFormat::I16 => { device.build_input_stream( &config.config(), move |data: &[i16], _: &_| { // do something with i16 samples }, err_fn, None, )? } sample_format => { return Err(format!("Unsupported sample format: {sample_format:?}").into()); } };

Note that we don't control when that callback fires. Once the stream is live, the audio system calls us whenever a new buffer of samples arrives.

Two Loops, One Channel

The UI side runs on its own schedule too. With egui and eframe, we supply a closure that gets invoked whenever a repaint is due:

eframe::run_simple_native( "spectrogram", NativeOptions { viewport: ViewportBuilder::default().with_inner_size(vec2(1100.0, 600.0)), ..Default::default() }, move |ctx, _frame| { // do something with ctx } ).unwrap();

So there are two independent event loops, neither of which we own. Answer: communicate via channels. At startup, we create a bounded channel with capacity 10 to avoid hoarding memory:

// Create channel for passing audio buffers const CHANNEL_CAPACITY: usize = 10; let (sender, receiver) = bounded::<(Vec<f32>, f32)>(CHANNEL_CAPACITY);

This uses crossbeam-channel, though the standard library's sync_channel offers something similar.

Inside the audio callback, we average the samples down to mono, accumulate them into a buffer, and send each full buffer through the Rust channel:

// Convert to mono and collect samples for chunk in data.chunks(channels) { let mono_sample = chunk.iter().sum::<f32>() / channels as f32; sample_buffer.push(mono_sample); // Send buffer when we have STEP_SIZE samples if sample_buffer.len() == STEP_SIZE { if sender .try_send((sample_buffer.clone(), sample_rate)) .is_err() { // Channel full, skip this buffer } sample_buffer.clear(); } }

A dedicated thread receives those buffers, applying a pre-calculated Hann window and then the FFT:

for (i, sample) in window.iter().enumerate() { windowed_samples[i] = sample * hann_window[i]; } let fft = windowed_samples[..].real_fft();

The FFT yields complex numbers, which we massage: first compute the magnitude (norm), then square it to get power:

let magnitude = fft[i].norm();

Normalize by the number of frequency bins to keep values in the 0–1 range:

let normalized_power = power / (WINDOW_SIZE as f32 * WINDOW_SIZE as f32);

Compensate for the Hann window's edge tapering, which we've already measured as a factor below 1, so dividing by it actually boosts the result:

let corrected = normalized_power / power_gain;

Then, for every bin except the DC offset (bin zero), double the value:

// Double power for one-sided spectrum (except DC bin) let final_power = if i == 0 { corrected // DC bin: don't double } else { 2.0 * corrected // All other bins: double };

Finally, convert to decibels with a 10x factor:

// Convert to dB (10*log10 for power) let db_value = 10.0 * (final_power.max(1e-20)).log10(); fft_magnitudes.push(db_value);

The resulting magnitudes are what the UI needs. Rather than another channel, we use a shared, mutex-protected VecDeque. On the UI side, we force continuous repaint b requesting one at the start of every callback:

eframe::run_simple_native( "spectrogram", Default::default(), move |ctx, _frame| { ctx.request_repaint(); // 👈 // drawing happens here } ).unwrap();

The UI thread then checks the mutex and pops any ready FFT results:

type Model = Arc<Mutex<ModelInner>>; #[derive(Default)] struct ModelInner { ffts: VecDeque<Vec<f32>>, sample_rate: f32, }

The only shared state besides the deque is the sample rate, since consumer hardware can't pick between 44.1 kHz and 48 kHz.

Data flow: the cpal callback pushes raw samples into a channel → the FFT thread pulls them, does the heavy math, and pushes results into the shared VecDeque → the UI pops from that deque when drawing.

But this design isn't perfect. If the window is minimized and drawing stalls while audio keeps coming, the VecDeque grows without bound—a potential memory blow-up. A simple fix: refuse to push if the deque exceeds a certain size.

Texture Management

Most of the UI is reconstructed each frame, as immediate-mode GUIs like egui naturally do. Drawing the spectrogram pixel-by-pixel via rectangles would be wasteful, though. Instead, we maintain a texture that we update incrementally.

Outside the repaint loop, we set up:

let mut texture: Option<egui::TextureHandle> = None; let size = [3700, 2048]; let mut img = egui::ColorImage::filled(size, Color32::BLACK); let mut state = State { model, index: 0, buffer: None, };

…a texture handle (initially None), a placeholder image, and a state object holding the model shared with the FFT thread. Inside the paint callback, we lazily allocate the texture handle:

let texture = texture.get_or_insert_with(|| { ctx.load_texture( "spectrogram", egui::ColorImage::filled([1, 1], Color32::WHITE), Default::default(), ) });

Then draw into an image and push it to the GPU:

redraw(&mut img.pixels[..], size[0], size[1], &mut state); texture.set(img.clone(), Default::default());

Finally, the texture is rendered with ui.image():

// ✂️ cut: more UI code ui.horizontal(|ui| { egui::Frame::default() .stroke(Stroke::new(1.0, Color32::DARK_GRAY)) .show(ui, |ui| { ui.image((texture.id(), texture.size_vec2() * 0.25)); }); }) // ✂️ cut: even more UI code

egui supports set_partial to update only a region of the texture, which would be smarter for one-column-at-a-time updates—but the full-image copy is fast enough for this machine. The 0.25 scale factor on texture size accounts for a high-DPI display; the canvas is effectively 3700×2048, roughly 4x the visible area. It runs in real time, albeit as a CPU hog.

Profile Performance

Why the high CPU usage? Time to fire up Apple's Instruments with Processor Trace support. Enabling debug info for the release build is all it takes:

# in `Cargo.toml` # ✂️ cut: package info, dependencies, etc. [profile.release] debug = 1 split-debuginfo = "packed"

The split-debuginfo setting generates the dSYM bundles Instruments needs. After cargo build --release, sign the binary with an entitlement file containing:

󰗀<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>com.apple.security.get-task-allow</key> <true/> </dict> </plist>

Applied with:

$ codesign -s - -f --entitlements ~/hello.entitlements target/release/spectrogram

The processor trace profile (even a one-second recording is heavy) shows exactly where cycles go: 25% of CPU time is spent cloning ColorImage:

texture.set(img.clone(), Default::default());

And 40% of total time goes into uploading textures from CPU to GPU—through glTexImage2D, which macOS implements via AppleMetalOpenGLRenderer:

The same screenshot with glTexImage2D highlighted

The profile also reveals egui is tessellating shapes on the CPU (transforming them into polygons for the GPU), which is expected.

The audio thread is busy too: 27% of its time goes to FFT computation, and separate com.apple.audio.IOThread.client threads each spend about 7% cloning sample buffers and roughly another 7% in try_send. None of that channel traffic is free.

What a Spectrogram Reveals

Once the spectrogram is running, it quickly becomes a tool for seeing music in a new way. Certain techniques and production choices jump out immediately when you can watch the frequency content of a recording unfold over time.

Vocals with strong vibrato, for example, produce visible wavelike patterns in the higher frequencies. Aretha Franklin’s “This Bitter Earth” is a great demonstration of that effect. Electronic music that uses frequency sweeps makes them plainly visible as diagonal lines, as in the opening of Madeon’s “Icarus.”

Chiptune tracks are always visually engaging — Meganeko’s “Discovery” is a good example. The tightly controlled synthesis of pop music shows up too: Sub Urban’s “Cradles” begins with mostly midrange content, then expands suddenly when the drums and vocals enter. Compare that with a 1954 recording of Line Renaud singing “Je ne sais pas,” which carries background noise in the low end long before the bass arrives.

Dense productions like Radiohead’s “Nude” make the spectrogram busy, reflecting how much is layered into the mix. Even the barely audible opening moments of Pink Floyd’s “Shine On You Crazy Diamond” draw visible shapes. Metal music often sits at the opposite extreme, packing energy across the whole frequency range — Atreyu’s “Bleeding Mascara” shows how much information can be packed in, even if it makes individual elements harder to separate visually.

A brown-ish bird, very cute.

Carolina Wren, Clear Lake, Houston, Texas

Dan Pancamo

For a cleaner example of vocal precision, Linda Ronstadt’s performance on “When You Wish Upon A Star” is worth a look. Speech also produces interesting patterns — James’s voice on an episode of Self-Directed Research shows how different spoken language looks compared to singing.

A small songbird perches on a twisted wire cable against a soft, blurred green background. The bird has a distinctive black head and throat, white cheeks, and a pale yellow-buff colored belly and underparts. Its wings show darker markings, and it has a short, conical beak typical of seed-eating birds. The bird's feet are wrapped around the metal cable as it maintains its balance. The shallow depth of field creates an attractive bokeh effect in the background, highlighting the bird as the main subject of the photograph.

Morelet's seedeater (Sporophila morelleti morelleti) Orange Walk, Belize

Charles J. Sharp

Spectrograms are also commonly used to study animal sounds, especially bird calls. A Carolina wren and a Morelet’s seedeater each produce distinct visual signatures that make identification possible at a glance.

Notes on Running It Yourself

The spectrogram was built for enjoyment rather than maximum performance, so there is room for optimization — but that’s not really the point. If you want to try it, patrons and sponsors of any tier can compile and run the program locally and ask for help on the Discord if needed.

On macOS, free alternatives to Loopback such as Blackhole exist, though they may require more setup. In a pinch, holding a phone up to the computer’s microphone will work, at the cost of some precision.