Why Media Files Break the Data-Pipeline Mold
At a fundamental level, audio is just wiggly air. A podcast conversation and an acoustic guitar track are both sequences of compressed air vibrations that we interpret through our ears. Once that sound is captured, encoded, and stored on a machine, however, it becomes a collection of relatively large binary files that look nearly identical to a computer scanning raw data. Teaching software to identify what that data represents — voices, beat patterns, individual frequencies — is hard enough. Doing it across the more than 60 million tracks in Spotify's catalog, with roughly 40,000 new songs added each day, is a problem of scale that most standard data tools are not built to handle.
Processing this kind of heavy binary media actually requires a different class of infrastructure than typical map-filter-reduce jobs. It requires tight integration between research tools and production engineering, support for Python (the dominant language in both audio processing and machine learning), and the ability to handle large-file I/O with both streaming and batch workflows. This is exactly the gap Spotify's open-sourced framework Klio was designed to fill.
Speaking a Common Language
Prior to Klio, Spotify's engineering teams had already built a scalable data-pipeline framework called Scio, which was extremely effective for jobs like the large data queries and ML models that fuel Discover Weekly, Release Radar, and other personalized playlists. But Scio's Java-based foundation created a collision course with the research teams, who were primarily working in Python. The standard workflow was slow, serialized, and siloed: researchers would develop an approach in Python, then hand it over to engineers who had to rewrite that work in Java before it could become part of a shipped product. These bespoke pipelines were also highly customized for individual teams, making them difficult to maintain.
The solution surfaced in 2019 when an ad hoc team of data engineers, ML researchers, and audio experts sat down to define requirements for a new media-processing framework. Early on, they concluded that Python was the only real choice for getting both groups to speak the same language without a translation layer. The result is a framework that lets researchers focus on experimentation and lets engineers focus on writing clean production code, without forcing anyone into a second language.
Inside Klio and Its Refined Beam Experience
At its core, Klio is an open-source framework for building data pipelines for audio and other binary files, based on Python and Apache Beam. It offers a “streamlined” take on Beam, with the stated goal of being more Python-native and more ergonomic than the stock experience. Some of its key architectural features include:
Streamlined Apache Beam: a Python-first wrapper around Beam that is designed to be accessible for researchers, avoiding the need for boilerplate code.
Open graph of job dependencies: supports top-down and bottom-up execution, which makes it easier to run and debug interconnected jobs.
Containerization of dependencies: allows users to package various native media dependencies, such as
libsndfileandffmpeg, into reproducible deployments.Heavy file I/O focus: built specifically for transforming large binary formats, not just lightweight tabular data.
Batch and streaming pipelines: designed to process in a continuous fashion or in large, discrete jobs.
The framework takes an opinionated stance on media workflows, pushing users toward writing smaller pipelines that do one thing well — such as extracting a beat track or measuring loudness — instead of giant conglomerations. These single-purpose blocks can then be chained together into larger graphs without rewriting existing jobs.
Although Klio provides a more structured way of doing things by default, users who need to step outside those guardrails can still drop into core Python Beam at any time.
Scale, Speed, and No Infra Teams Required
The practical payoff is measurable. A key benchmark for the Klio development team came when they used the new framework to downsample every track in Spotify's 60-million-song catalog. Downsampling is often the first step in audio analysis, and the dataset involved well over 100 million files when counting multiple other releases of the same songs. Prior to building Klio, Spotify's fastest time on this massive job clocked in at roughly three to four weeks. With Klio, the run was completed in six days, bringing a four-times reduction in cost along with the speedup.
Much of the efficiency gain can be attributed to the framework's general resistance to duplicate work — it intentionally avoids processing audio files that already have output for a specific task. The clean, single-purpose building blocks created by Klio users further reinforce this “don't repeat yourself” principle.
Klio is configured to run out-of-the-box on Google Cloud Platform, but it was designed to be cloud-agnostic. Thanks to its underlying Apache Beam foundation, work has already been done successfully testing Klio on Amazon AWS and S3, using Klio's Direct Runner. Some limitations exist due to current constraints in Beam Python with certain engines like Apache Flink and Spark, noting in particular that not all features can be used on every engine. However, Apache Beam’s continued extension of compatibility support is expected to soften these boundaries over time.
By default, when running in the cloud, Klio connects to managed processing engines that handle autoscaling for production pipelines. This allowance for infrastructure as a service means engineers can keep their focus on application-level traits like monitoring data and growing the pipeline, rather than constantly tuning heavy workloads. Meanwhile, reproducibility and scalability come from standard cloud infrastructure features and containerization tools, removing the burden of building bespoke systems from the ground up.
Open Source and the Unexpected Use Cases
Klio started as an internal proof of concept less than two years ago, built out of necessity to address challenges Spotify was facing. From the start, though, the intent was to release it as free and open source software. The rationale echoes the company's experience with Backstage, its open platform for developer portals: Spotify sees value in open sourcing its infrastructure, and the engineering community benefits when tools are shared.
Releasing Klio means others can now run the same kind of large-scale media processing that Spotify has been doing for close to a decade. That includes extracting and transforming audio from tracks in the catalog on weekly, daily, and streaming bases. Those pipelines power the Audio Features API that fingerprints songs by unique attributes, in-house review tools, and market-specific features such as the Sing Along feature in Japan, which separates vocals from instruments as tracks are uploaded to create interactive versions.
The broader hope is that open sourcing follows the pattern of Backstage: the community produces use cases that were never anticipated internally. Because Klio enables heavy-duty media processing at scale for anyone—not just large tech companies—the team is particularly interested in what academics and research institutions build with it. The ability to process non-music audio, from dolphin sounds to other research data, is part of that unknown territory.
The project owes thanks to the Klio team, contributors over the years, and the sibling framework Scio. While Klio is a product that only Spotify could have built given its audio infrastructure, the real value will come from external contributions and the new problems the community decides to solve with it. Documentation is available at docs.klio.io to get started.
