Turning Recorded Performances into Editable MIDI

MIDI has been the backbone of digital music production for decades, functioning as a digital score that computers can read and edit. But getting audio into MIDI has traditionally been a bottleneck: musicians typically need dedicated MIDI hardware or must painstakingly program notes by hand. Live performances are especially hard to parse — once recorded, the individual notes blend together in ways that are difficult for software to separate.

Spotify's Audio Intelligence Lab, working with Soundtrap, set out to solve this with Basic Pitch, an open source machine learning tool that transcribes audio into MIDI. It accepts recordings from a wide range of instruments, including the human voice, and produces MIDI output that can be dropped into any digital audio workstation for editing. The model is available as an open source release, and a browser-based demo runs at basicpitch.io with no installation required.

What Sets Basic Pitch Apart

Basic Pitch was designed around three properties that distinguish it from earlier transcription systems:

  • Polyphonic and instrument-agnostic: Unlike many note-detection tools that handle only one note at a time or a single instrument type, Basic Pitch identifies multiple simultaneous notes across instruments as varied as piano, guitar, voice, and ocarina.
  • Pitch bend detection: Expressive techniques like vibrato, glissando, and slides are captured rather than flattened away, preserving nuance that typically gets lost in audio-to-MIDI conversion.
  • Lightweight speed: The model is compact enough to run faster than real time on most modern hardware, as detailed in the accompanying ICASSP 2022 paper.

By combining these capabilities, Basic Pitch lets musicians capture ideas in the moment with whatever instrument they have on hand, then work with those ideas in MIDI form without needing specialized gear or manual note entry.

Small Model, Not Small Ambitions

Common wisdom in machine learning holds that bigger models produce better results. Many state-of-the-art systems today lean on billions of parameters to achieve high accuracy on narrow tasks. But the Spotify team wanted a model that handled many instruments and polyphonic input — a tool for both piano virtuosos and casual vocalists — without inheriting the computational weight of a mega-model.

Music transcription poses challenges that push back against the "bigger is better" approach. In polyphonic passages, notes overlap in both time and frequency, making them hard to untangle. It is also ambiguous where one note ends and another begins: a singer gliding through "sooooong" might be heard as two notes, three, or five depending on the listener. A useful system must make reasonable judgments across such gray areas for any instrument, from kazoo to soprano opera.

Architecture Choices for Efficiency

Basic Pitch keeps its footprint small through several techniques drawn from prior research in audio intelligence:

  • A harmonic constant-Q transform as the input representation
  • Joint modeling of note onsets, frames, and multipitch information
  • Fewer layers and fewer parameters than conventional deep models

The resulting shallow architecture is remarkably light: under 20 MB of peak memory and fewer than 17,000 parameters. For context, that is 17 thousand, not 17 billion. Despite the modest size, benchmarking across datasets of different instrument types — including the Molina vocal dataset, a particularly tricky case — shows accuracy comparable to dedicated single-instrument systems.

Try It Yourself

Basic Pitch can be tested directly in a web browser at basicpitch.io. For developers, the model is open sourced on GitHub. The technical details are documented in the ICASSP 2022 paper, and a video of that presentation is also available.

From Soundtrap to the Public Domain

Basic Pitch began as an internal project aimed squarely at Soundtrap users. The goal was to give audio producers a fast, accurate way to convert polyphonic audio into MIDI without needing specialized hardware or manual transcription. Once the model proved itself internally, Spotify opted to release it broadly—both as an open source library and a free online tool—so that musicians, developers, and researchers could all build on it.

The practical use cases go beyond simple transcription. A guitarist, for instance, could hum or play a melody into a microphone and get a clean MIDI file to drop into a DAW. But the model’s design also opens the door to more ambitious integrations, such as real-time systems that listen to a live performance and trigger accompanying MIDI instruments in reaction to what is being played.

The Case for Smaller Models

Spotify’s decision to share Basic Pitch is not just about the tool itself; it is also a statement about model design philosophy. Heavy, computationally expensive models are sometimes the only viable option for a given task, but Spotify argues they should not be the default. By publishing this lightweight approach, the team hopes to encourage other ML researchers to consider whether a leaner model might be sufficient for their own projects.

The model was detailed in a paper presented at the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing, which described it as a lightweight, instrument-agnostic solution for polyphonic note transcription and multipitch estimation. It also drew on prior work in melody extraction and f0 estimation, including research on harmonic convolutional neural networks and deep salience representations for pitch detection in polyphonic music.

Open Source and Ready for Extension

The library ships under a standard open source license, meaning it can be incorporated into other music production tools or modified for entirely new purposes. An early example of that flexibility is NeuralNote, a free VST/AU plugin that pairs Basic Pitch’s audio-to-MIDI capabilities with a user’s favorite DAW. Spotify is inviting feedback and contributions through the official GitHub repository, acknowledging that real-world use will inevitably surface areas for improvement that internal testing could not catch.

The initial release is best understood as a starting point for transcription workflows—not a finished product. The team expects to learn from how the model performs in the wild and to discover new possibilities for its use as developers push it in directions Spotify has not yet considered.

Contributors to Basic Pitch include Rachel Bittner, Juanjo Bosch, Vincent Degroote, Brian Dombrowski, Simon Durand, Sebastian Ewert, Gabriel Meseguer Brocal, Nicola Montecchio, Adam Rackis, David Rubinstein, Ching Sung, Scott Sheffield, Peter Sobot, Daniel Stoller, Charae Tongg, and Jan Van Balen.