Cryptographic Monitoring With Zero Sampling at Meta Scale

FBCrypto, Meta's managed cryptographic library, sits in the majority of the company's core infrastructure services. That level of adoption makes the library's health critical—and makes understanding exactly how it's used across thousands of services a serious engineering challenge.

Meta's answer is a cryptographic monitoring system built around aggregation and buffering rather than per-event logging. The payoff: engineers get a complete, unsampled picture of every cryptographic operation on the fleet, with negligible compute overhead. That visibility lets Meta detect weak algorithms before they're exploited, rotate keys before they're overused, and prepare for post-quantum cryptography migrations.

Why Log Every Crypto Operation?

There's a hard limit to how much data any single symmetric key can safely protect. Logging every use of a key lets Meta detect overuse and rotate keys proactively. The same logs build an inventory of all cryptographic usage, so engineers can identify call sites using weakened algorithms and migrate them before those algorithms become liabilities—a task that becomes urgent when a vulnerability in a primitive is discovered.

More recently, this monitoring has become central to Meta's post-quantum readiness work. Having complete data on asymmetric cryptography usage informs prioritization decisions for migrating quantum-vulnerable use cases.

The monitoring data also acts as a proxy for client health. With no sampling, engineers see exact call volumes and success rates, making it possible to detect anomalous drops during large-scale code migrations. The dataset includes library versioning information, so rollout teams can see in real-time which clients have adopted the latest changes.

Sampling Wasn't the Answer

Meta's standard logging stack is Scribe, with data persisting to Scuba for short-term storage and Hive for long-term analytics. Typically, engineers call the Scribe API directly to log each event. For FBCrypto, that would mean constructing a log entry for nearly every cryptographic operation across the fleet. At the scale Meta operates—roughly 0.05% of all CPU cycles are spent on X25519 key exchange alone—that approach would consume unreasonable write throughput and storage.

Sampling (logging only 1 in X operations) would solve the capacity problem but was a non-starter. Sparse logs would give an incomplete picture of library usage, undermining the entire monitoring effort. Instead, FBCrypto uses a buffering-and-flushing strategy: cryptographic events are aggregated in memory for a preconfigured interval, then exported to the data store in a single flush.

During aggregation, each unique event maintains a count. When flushing, that count is exported alongside the log, indicating how frequently the event occurred.

In the illustration, the key myKeyName is used five times for AES-GCM-SIV encryption, and the log records a count of five. Since individual machines can compute millions of cryptographic operations daily, this aggregation produces significant compute and storage savings.

How the Buffered Logger Works

The aggregation and flushing logic lives entirely inside FBCrypto, on the client hosts. When a client calls an operation like encrypt(), the operation executes and the event is added to an aggregated buffer held by what Meta calls the buffered logger. The logging is transparent—the FBCrypto interface doesn't change, so clients don't need to know logging exists.

In multithreaded environments, all threads log to the same buffer, requiring a data structure that's performant under heavy concurrent writes. A background thread on each host periodically calls the Scribe API to export the buffered logs and flush the map's contents.

Key Optimizations

Making this work across Facebook, WhatsApp, Instagram, and other major products required additional design decisions.

Partially Randomized Flushing

The buffering strategy created a problem: when large job restarts affected many machines simultaneously, their logs flushed at roughly the same time, causing "spiky" writes to the logging platform followed by underutilization. To smooth this out, Meta applies a randomized delay on a per-host basis before the first log flush. This distributes write spikes across time and creates a more uniform load on Scribe.

Derived Crypto Aggregation

FBCrypto supports derived crypto, where "child" keysets are derived from "parent" keysets using a key derivation function (KDF) with a salt. Some large-scale use cases generate millions of derived keys. Logging each derived keyset as its own row consumed massive space and overloaded backend data stores.

Meta now aggregates cryptographic operations for derived keys under the parent key's name. This cuts the vast majority of logging volume and still detects key overuse—in the worst case, the parent-key aggregation serves as a pessimistic counter for any individual child key.

The Folly Library

The buffered logger uses Meta's folly::ConcurrentHashMap, designed for heavy writes in multithreaded environments with guaranteed atomic access. This choice keeps concurrent logging performant without locking bottlenecks.

The Value of Unified Infrastructure

Meta's emphasis on unified infrastructure made this monitoring system practical. Most machines in the fleet can already log to Scribe, so log ingestion required no new infrastructure. FBCrypto's broad adoption means the monitoring benefits apply fleet-wide without requiring clients to migrate to a new library or API.

This avoids the fragmentation that would force multiple custom solutions, each with its own engineering and maintenance burden.

Impact Across Security and Reliability

Preemptive Vulnerability Mitigation

Long data retention enables trend monitoring and predictive analysis. Cryptography experts can analyze the data and identify clients using cryptography in risky ways before those risks become actual vulnerabilities. This advances Meta's ability to define and enforce clear cryptographic maturity standards across all systems—something not previously possible at this scale.

For post-quantum cryptography readiness, this is particularly relevant. Organizations must find clients using quantum-vulnerable algorithms and migrate them. Early detection means affected teams have ample time to integrate migration work into their roadmaps, making cross-team collaboration smoother.

Reinforcing Infrastructure Reliability

The unsampled dataset offers precise insight into client health. During large migrations, detectors and alarms built on the monitoring data catch anomalous drops in success rate or call volume, often indicating a bug in new code paths. The library versioning information enables real-time tracking of feature rollouts—engineers can see exactly which clients have adopted the latest changes and move with confidence through fleet-wide migrations.

Cryptographic logging at Meta has not been without operational headaches. Several recurring issues stand out.

Scribe and Scuba load spikes

Even with aggressive optimizations, crypto usage occasionally outpaces projections, leading to unexpected write pressure on Scribe. The engineering team handles this in direct coordination with the Scribe owners, and the earlier design decisions make absorbing the extra throughput comparatively straightforward.

Real-time analytics via Scuba is another pressure point. Scuba is built for warm data and gets inefficient as datasets grow. To keep compute costs manageable, the team pushes longer-term records into Hive tables for cold storage, reserving Scuba for genuinely time-sensitive questions.

Shutdown edge cases

Clients flush their log buffers on a timed interval, but also perform a final flush whenever a job is shutting down. That final flush happens inside a “shutdown environment,” which creates awkward conditions for reaching Scribe and its dependencies. Most complications trace back to nuances of folly::Singleton, Meta’s standard singleton management library. In the Java world, a shutdown-time flush demands strictly synchronous I/O and fast execution, leaving little room for error.

What’s next for crypto monitoring

The current system works, but the roadmap includes several concrete improvements. One immediate goal is improving Scribe throughput and the efficiency of Scuba storage so the infrastructure footprint stays reasonable.

On the security side, the team continues to mine logging data to identify use cases that would break in a post-quantum cryptography (PQC) world, then migrates them to algorithms and configurations that can withstand quantum attacks. On the reliability side, the focus is developing a clearer picture of end-to-end latency across cryptographic operations.

There is also the longer-term push for uniformity. FBCrypto offers a single, unified API surface, but not every crypto use case across Meta uses it. Several teams still rely on separate telemetry and data collection paths, and closing that gap requires more than trivial engineering effort.

Acknowledgments

Grace Wu, Ilya Maykov, Srinivas Murri, Isaac Elbaz, and the rest of Meta’s CryptoEng team made this work possible.