Rethinking an Observability Pipeline on Cloudflare
When Baselime joined Cloudflare in April 2024, its architecture had grown into a sprawling network of hundreds of AWS Lambda functions, dozens of databases, and a similar number of queues. That complexity carried a heavy operational toll and a rapidly inflating cloud bill. Rebuilding on the Cloudflare Developer Platform has since cut estimated compute costs by over 80% for the standalone Baselime offering (based on the Workers Paid plan), while also simplifying the system and improving latency, scalability, and reliability. The first week after the acquisition announcement saw usage surge past a billion events daily, with weekly active users tripling, which accelerated the need for a more sustainable foundation.
The Old AWS Stack: Powerful but Painful
The previous data pipeline—responsible for ingesting, processing, and storing tens of billions of events daily—was built on AWS Lambda, Cloudfront, Kinesis, EC2, DynamoDB, ECS, and ElastiCache. The core flow moved data through several distinct stages:
- Data receptors received telemetry from sources like OpenTelemetry, Cloudflare Logpush, CloudWatch, and Vercel. These endpoints handled validation, authentication, and normalization into a common internal format, running on either AWS Lambda (with function URLs behind Cloudfront) or ECS Fargate.
- Kinesis Data Stream transported data between receptors and the processor.
- The processor, a single Lambda function, enriched data for storage and performed real-time error tracking and log pattern detection.
- A self-hosted ClickHouse cluster on EC2 handled all indexing and storage.
Beyond these components, the stack also relied on Firehose, S3, SQS, DynamoDB, and RDS for error handling, retries, and metadata. This complexity became a major liability at scale. Retries between receptors and Kinesis demanded careful orchestration across multiple services, while self-hosting ClickHouse meant constant capacity planning and infrastructure upkeep.
Costs grew unpredictably across Lambda, Kinesis, and EC2, with quieter culprits in Cloudfront (needed for custom domains on Lambda URLs) and DynamoDB. The biggest hidden expense was I/O wait time: moving data between stages repeatedly blocked on network requests, consuming over 70% of wall time in Lambda functions. The entire deployment also lived in a single AWS region, eu-west-1, causing high latency for developers outside continental Europe.
New Architecture: Workers at the Core
The shift to the Cloudflare Developer Platform allowed a complete rethink around globally distributed, low-latency primitives. Workers now power every data receptor and the processor. With minimal cold starts and global default deployment, developers sending logs and traces see consistently lower latency. The internal communication between pipeline steps now relies on JavaScript-native RPC, which is lightweight, low-latency, and lets separate components behave like functions within a single process rather than distinct applications.
Workers also eliminated the need for custom rate-limiting infrastructure. The built-in Rate Limiting binding replaces what previously required a combination of DynamoDB and ElastiCache. Extensively using ctx.waitUntil moves data transformation out of the request/response path, shaving further latency off receptor calls.
Durable Objects for Stateful Processing
Durable Objects bring stateful capabilities to a serverless environment. In the data pipeline, they manage real-time error tracking and log pattern detection. For error tracking, a durable object is created per error type, handling frequency tracking, customer notifications, and notification channels. This single building block replaces what previously needed ElastiCache, Kinesis, and multiple Lambda functions all coordinating to protect an RDS database from high-frequency errors. Durable Objects also handle alerting: the alarm handler covers what previously required orchestration across EventBridge Scheduler, SQS, DynamoDB, and several Lambda functions.
Workers Analytics Engine Replaces ClickHouse
Managing a self-hosted ClickHouse cluster diverted time away from product development. Workers Analytics Engine provides a scalable, high-cardinality analytics database that accepts synchronous event writes. The team built on the same underlying technology, and also made internal changes to Workers Analytics Engine to natively enable high dimensionality in addition to high cardinality.
Both systems leverage Cloudflare's ABR (Adaptive Bit Rate) analytics, which stores telemetry data across multiple tables at resolutions from 100% down to 0.0001%. Queries against the lower-resolution tables run orders of magnitude faster, at the cost of some accuracy. The analytics engine dynamically selects the appropriate table per query to optimize speed while returning the most accurate result for any dataset size or timeframe. The previous system always queried the full dataset, so the new approach delivers faster queries across the entire user base.
Supporting services round out the platform: Queues provides asynchronous messaging and decoupling; D1 serves as the transactional database for queries, alerts, dashboards, and configurations; Workers KV handles fast distributed storage; and the Hono framework powers all APIs.
Migrating with the Strangler Fig Pattern
Baselime's event-driven architecture records every user action as an event. The migration moved this from AWS EventBridge and SQS entirely onto Cloudflare Queues, using the strangler fig pattern to replace components incrementally without disrupting uptime or data consistency. Early on, a central Cloudflare Queue became the backbone for all transactional event processing, with every event—user signups, dashboard edits, anything—funneled into it and dynamically routed to the relevant application parts. User actions were synced into both D1 and KV, mirroring all activity across AWS and Cloudflare during the transition to guarantee no data loss.
The data pipeline itself was migrated outside-in. Data receptors moved first, with telemetry written simultaneously to the AWS ClickHouse cluster and to Workers Analytics Engine for the full 30-day retention period. The final step was rewriting all API endpoints from AWS Lambda and ECS into Cloudflare Workers, then simply switching DNS records to point at the new Workers.
The entire process—from data pipeline to all API rewrites—took a team of three engineers less than three months.
Costs fell across the ingestion and storage tiers
Baselime’s migration to Cloudflare produced the biggest savings in two areas: the data receptors that ingest telemetry and the ClickHouse cluster that stored it. Both were swapped for Cloudflare services in 2024, and the monthly effect on AWS spending was substantial.
Data receptors: from Lambda to Workers
The data receptors moved off AWS Lambda in early June 2024. Lambda costs dropped by over 85%, mostly because the receptors were no longer paying for I/O time spent pushing data into a Kinesis Data Stream in the same region.
Figure 4: Baselime daily AWS Lambda cost [note: the gap in data is the result of AWS Cost Explorer losing data when the parent organization of the cloud accounts was changed.]
The receptors had previously used CloudFront to serve custom domains. Once they ran on Cloudflare, that dependency disappeared and CloudFront costs went to $0.
Figure 5: Baselime daily Cloudfront cost [note: the gap in data is the result of AWS Cost Explorer losing data when the parent organization of the cloud accounts was changed.]
Baselime estimates that, as a regular Cloudflare customer, the Workers bill for the receptors would be about $25/day, versus $790/day on AWS — a reduction of more than 95%. The pricing model explains most of the gap: Workers bill for CPU time, and the receptors are largely I/O-bound data movers.
ClickHouse: from self-hosted to Workers Analytics Engine
Replacing the self-hosted ClickHouse deployment with Workers Analytics Engine meant accounting for more than EC2 instances. Disk, networking, and the Kinesis Data Stream all factored into the comparison. Baselime completed that switch in late August and saw reductions above 95% in both Kinesis and EC2-related costs.
Figure 6: Baselime daily Kinesis Data Stream cost [note: the gap in data is the result of AWS Cost Explorer losing data when the parent organization of the cloud accounts was changed.]
Figure 7: Baselime daily EC2 cost [note: the gap in data is the result of AWS Cost Explorer losing data when the parent organization of the cloud accounts was changed.]
As a regular Cloudflare customer, Baselime estimates the Analytics Engine bill would land around $300/day after the migration, compared with $1150/day on AWS — over 70% less.
Lower cost came with faster queries and less operational burden
The move also improved performance. Event ingestion now happens across Cloudflare’s network, closer to users, and query responses are faster thanks to Cloudflare’s experience operating ClickHouse at scale. Throughput limits are no longer a constraint: when Workers Logs launched on September 26, 2024, the system was already handling a higher volume of events without sacrificing speed or reliability.
The headline numbers also exclude total cost of ownership. Baselime simplified its architecture and codebase significantly, since the platform now handles more of the infrastructure. The team gets paged less, spends less time watching infrastructure, and has more room for product work.
Outlook
Baselime now runs on Workers, Durable Objects, and Workers Analytics Engine as a fully serverless, globally distributed system. The company plans to deliver real-time error tracking, alerting, and a query builder for high-cardinality events directly in the Cloudflare Dashboard by early 2025.
