Storage benchmarking that reflects real production load

Synthetic I/O benchmarks traditionally focus on pushing drives to their absolute limits, with the winner being the device that sustains the highest IOPS or throughput. At Meta, however, production applications rarely approach those peaks. Instead, they are highly sensitive to latency outliers. FioSynth is an open-source benchmark tool designed around that reality: it automates the execution of storage workload suites derived from production I/O traces and parses the results into a form that is easy to compare and interpret.

The tool's workload suites are rate-limited to match I/O patterns commonly observed on Meta's hardware. The most important metric is not peak performance but outlier latency—P99, P99.99, P99.9999, and beyond—while sustaining the expected I/O rates. Drives that maintain the lowest tail latency at those rates are deemed the best performers, regardless of their headline IOPS or throughput numbers.

How it works

FioSynth is built on top of fio, a flexible I/O workload generator. It is designed to be easy to deploy on a wide range of storage devices and configurations, and to produce results that can be viewed without heavy tooling. Because the workloads are scalable on a per-terabyte basis, engineers can directly compare direct-attached storage with disaggregated storage (iSCSI, AOE, etc.), small drives with large devices, and non-RAID with RAID configurations.

Key features include:

  • Dozens of workload suites for both flash and hard drives, capturing the I/O profiles of a diverse set of services.
  • Workload scaling based on available storage capacity, allowing apples-to-apples comparisons across devices of differing sizes.
  • A result parser that summarizes each workload suite into a single .csv file.
  • Support for client/server mode, where all workload suites run in parallel across multiple hosts; results are recorded per-server in individual .csv files, while a combined summary is written to a separate .csv file.
  • Optional device health log collection before and after each workload, enabling write amplification factor (the ratio of data physically written to flash media versus data written by the host) to be calculated per workload.
  • Definition files that separate preconditioning workloads from measurement workloads. Flash drives are preconditioned by default before performance measurement, but this step can be skipped via a command-line option.
  • Configurable run cycles, set either via the suite definition file or command line, to ensure repeatability of results.

Why it matters

FioSynth has been used internally at Meta for years to catch performance inefficiencies in flash devices before they reach the data center and to reproduce functional issues first seen in production. Several flash device suppliers have also used the tool to tune their products to meet the performance targets specified in the Open Compute Project's NVMe Cloud SSD Specification.

Making FioSynth available to the open source community serves three goals: helping storage manufacturers optimize drives for hyperscale workloads, standardizing flash preconditioning methodology, and lowering the barrier to running and visualizing storage benchmark results.

The tool is available at https://github.com/facebookincubator/FioSynth.