An Adtech Pipeline That Grew Nearly 50x in Three Years

TripleLift, a programmatic advertising company, runs a data pipeline that aggregates and delivers reporting data from billions of daily events. The current system, which the company describes in detail based on circa-April 2020 statistics, processes 30 billion events per day in its largest event log alone.

The pipeline has scaled dramatically in recent years. From 2016 to 2017, data volume increased 4.75x, followed by 2.5x growth from 2017 to 2018, and another 3.75x from 2018 to 2019—amounting to nearly 50x growth over three years.

Why the System Was Built

TripleLift's engineering team needed a pipeline capable of:

  • Scaling with rapid data volume growth
  • Aggregating log-level data cost-efficiently
  • Maintaining a clear, manageable job dependency chain
  • Automating idempotent job runs and retries
  • Delivering data to BI and reporting tools within expected SLAs
  • Handling query load on those reporting tools

System Scale and Workload

  • The largest event log receives 30 billion events daily
  • 5 daily aggregation jobs, with the largest outputting 7.5 GB
  • 25 hourly aggregation jobs, with the largest outputting 2.5 GB
  • 15 hourly jobs ingest aggregate data into BI tools
  • The highest-cardinality, normalized aggregation has 75 dimensions and 55 metrics

The Kafka cluster feeding the pipeline has an approximate I/O of 2.5 GB/hour.

Architecture Overview

The pipeline runs batch processes in a four-stage flow: raw event collection and persistence, data normalization via multiple aggregation levels, persistence of aggregated data into datastores, and exposure of that data in UIs and reporting tools.

Step 1: Collection. Raw event data is published to 50+ Kafka topics. Secor, an open-source consumer from Pinterest, reads these events and writes them to AWS S3 in parquet format.

Step 2: Aggregation. Apache Airflow handles scheduling and dependency management. It kicks off aggregation tasks by submitting jobs to the Databricks API. Apache Spark runs these aggregations on Databricks clusters. The process begins with denormalization: raw event logs are joined into wide tables that capture the full picture of what happened before, during, and after an ad auction. These denormalized logs are stored in S3.

Once denormalization tasks succeed, Airflow triggers downstream normalization tasks. Each rolls denormalized data up into narrower sets of dimensions and metrics aligned with specific reporting contexts. These final aggregations are also persisted to S3.

Step 3: Persistence. When each final aggregation completes, Airflow starts downstream ingestion tasks. One copies aggregated data into Snowflake, which serves as the backend for BI tools. Another ingests data into Imply Druid, a managed cloud platform with a time-optimized, columnar datastore for ad-hoc analytics queries over large datasets.

Step 4: Querying. BI and data engineering teams jointly expose the data through internal reporting APIs, Looker (backed by Snowflake), and Imply Pivot, the drag-and-drop analytics UI bundled with Imply Druid.

Lessons Learned

The team found that data decisions have far-reaching consequences. Once a field's derivation logic is defined, changing it is difficult because historical continuity for that field often must be preserved. A bug left running across multiple aggregation levels can make the backfill for that period expensive and time-consuming. And if incorrect or incomplete data becomes queryable, there is no way to know where it has already propagated.

What They'd Do Differently

For a long time, TripleLift lacked a clear data-access strategy. No retention policy existed, so data was stored indefinitely, and users could query all of it—which degraded query performance for everyone else. The team also failed to distinguish internal versus external reporting and reporting versus analytics workloads. That lack of discipline shaped the data itself: there was no hot versus cold tiering, no distinction between high and low granularity reporting data, and no sensible retention policy.

TripleLift has since implemented AWS S3 lifecycle rules, defined retention policies per queryable datasource, and assigned reporting tools to either handle quick investigative queries or large reports over long date ranges—but not both.

Future Plans

The company plans to complement its batch pipeline with a real-time streaming application built on Kafka Streams. That technology was chosen from proofs of concept that also evaluated Spark Structured Streaming and KSQL.

Monitoring and Team

Application metrics are stored in Prometheus, then aggregated in Grafana dashboards with alerts configured on those dashboards.

The data engineering team consists of 4 engineers, about a tenth of the entire engineering organization. They work closely with infrastructure, solutions, and data science teams. For example, the team recently onboarded data science onto Airflow, and their model runs are now automated.

Development languages are Python for Airflow, Spark Scala for aggregations, and Java for some reporting tools.