Achieving observability in async workflows - Netflix TechBlog

Debugging Distributed Async Workflows Without the Headache

Asynchronous workflows are a cornerstone of modern backend architecture, but they come with a hidden cost: observability. When something goes wrong in a system with multiple moving parts powered by background jobs, pinpointing the failure can become a time-consuming puzzle. Logs are scattered, requests span multiple services, and understanding the root cause of an issue often requires manual cross-referencing of data from a production database and external systems.

This was the exact scenario faced by the engineering team at Netflix responsible for Prodicle Distribution, a service that handles watermarking and secure document delivery. Their journey to rebuild this service with a focus on operational clarity highlights the architectural decisions and platform investments required to make async workflows truly debuggable.

The Problem: Supporting a Production's Critical Documents

Prodicle is a suite of applications that connects Netflix Studio Engineering with worldwide content productions. A key tool within this suite is Prodicle Distribution, which enables a Production Office Coordinator to send confidential, watermarked documents to crew members. These can be delivered as attachments or via secure links, and tracking this delivery is a core requirement.

The service's demands are significant. A single distribution job for a show like La Casa De Papel could involve 10 files being sent to 20 recipients, resulting in 200 unique watermarked documents. The backend must handle bursty workloads, processing thousands of watermarks within a minute while maintaining low, non-linear latency. This involves integrating with Google Drive (to make watermarked PDF copies), adding password protection, creating revocable links, generating thumbnails, and sending notifications through email and mobile push. Core requirements include elasticity and the strong observability of this entire pipeline for both users and internal teams.

Initially, the Distribution feature ran as a standalone Ruby on Rails application. As adoption grew, the ownership team faced mounting complexity. The single-threaded nature of a traditional Rails app wasn't suited for handling large-scale, parallel watermarking tasks, and debugging meant traversing multiple codebases and observability systems. New feature requests compounded the maintenance burden, making a fresh approach necessary.

A Shift to a Purpose-Built Orchestrator

The team decided to rebuild the service in Java and set ambitious goals for near real-time performance. They needed a highly scalable service and an orchestrator with robust visual traces. Rather than building this infrastructure from scratch, they evaluated internal platform teams' offerings, including Netflix's Conductor and the newer Cosmos platform. Cosmos was originally designed for media processing, but after early prototypes and load tests, it proved capable of handling the generic, file-based workflows required by Prodicle Distribution.

Cosmos provided a compelling fit due to:

• Low latency variance through the system
• A clear separation of concerns between the API layer, workflow definitions, and individual functions
• Ease of load testing and an API layer that could be customized with notifications
• A powerful observability portal with strong search capabilities

This migration also allowed the team to delegate the ownership of the watermarking function to another dedicated internal team, allowing them to focus on the Distribution service's own features.

Easier On-Call and Increased Adoption

With the new Cosmos-based system, on-call engineers can now use the Nirvana observability portal to visualize the workflow state and trace data across the entire system. The ability to search through context-rich logs and see the flow of data visually has significantly simplified identifying and resolving bottlenecks.

As productions resumed amidst a global pandemic, the service saw a 10x increase in adoption between June 2020 and April 2021. After an incremental release starting in January 2021 and completing by April, the system now supports hundreds of productions, processing tens of thousands of distribution jobs and millions of watermarks each month.

Offloading the infrastructure and watermarking concerns to platform teams has not only reduced the number of support tickets but also freed up engineering cycles. The service is well-positioned for the future, with plans to expand into areas like watermarking for images and videos, without investing in building and operating the underlying async infrastructure. This collaboration provides a significant benefit: other teams can easily integrate services by exposing functions within Cosmos, further simplifying the overall Netflix ecosystem base architecture.