Why accurate time matters at scale
Nearly every internet-connected device carries an onboard clock, and most of those clocks drift by seconds per day due to inexpensive internal oscillators. Correcting that drift matters: distributed databases rely on precise timestamps to order transactions, and log aggregation across thousands of servers needs millisecond-level agreement. At Facebook's scale, these requirements compound — knowing the exact time delta between any two servers in a datacenter, and across datacenters, is critical for consistency.
Our legacy NTP deployment, based on the common ntpd daemon, delivered accuracy in the range of 10 milliseconds. That was no longer sufficient. We evaluated chrony, a modern NTP implementation with support for the extended interleaved protocol, and found it dramatically more precise — often by two orders of magnitude. After replacing ntpd with chrony across our fleet, we pushed typical synchronization error from roughly 10 ms down to about 100 microseconds. The same stack now powers our public NTP service at time.facebook.com.
The leap-second problem
Earth's rotation isn't perfectly regular, so occasionally a leap second is inserted or removed to keep civil time aligned with astronomical time. For applications that assume monotonic time, that single-second jump can cause dropped transactions or software faults. One mitigation is leap-second smearing: instead of stepping the clock by one second, the adjustment is distributed as tiny increments across many hours. In our architecture, smearing is applied at the stratum-2 layer so downstream servers never see a discontinuity.
Hierarchy of time sources
Our NTP deployment follows the standard stratum model:
- Stratum 0: GNSS satellites carrying atomic clocks (GPS, GLONASS, Galileo).
- Stratum 1: Local atomic clocks synchronized to GNSS signals.
- Stratum 2: NTP servers that sync to stratum 1 and apply leap-second smearing.
- Stratum 3: Larger pool of servers that receive smeared time and have no leap-second awareness.
Further strata can be added to distribute load; the depth depends on the precision and scale required. We tested two daemons for this role: ntpd, the long-standing reference implementation, and chrony, which offers extended NTP modes and tighter synchronization. Resource usage was comparable, though chrony consumed roughly 1 MiB less RAM.
Trusting the daemon's estimates
Both ntpd and chrony report estimated offsets and error bounds via their command-line tools. These numbers rest on assumptions that rarely hold perfectly:
- The network path between client and server is symmetric.
- Timestamps are added to a packet and dispatched immediately via
send(). - Oscillator temperature and supply voltage remain constant.
With ntpd's ntpq, a reported offset of 0.185 ms doesn't reflect reality — the actual error window is typically ten times larger, meaning the clock is probably within ±2 ms. Our measurements showed typical accuracy near 10 ms with ntpd.
[user@client ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
=========================================================================
+server1 .FB. 2 u 406 1024 377 0.029 0.149 0.035
+server2 .FB. 2 u 548 1024 377 0.078 0.035 0.083
*server3 .FB. 2 u 460 1024 377 0.049 -0.185 0.114Switching to chrony changes that picture. The output below shows the measured offset, the original measurement adjusted for slews, and the estimated maximum error. When chrony peers with another chrony instance, it uses the extended NTP protocol, which tightens precision by roughly 100x. The estimated error drops from tens of milliseconds to hundreds of microseconds.
[user@client ~]# chronyc sources -v
210 Number of sources = 19
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| / '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^+ server1 2 6 377 2 +10us[ +10us] +/- 481us
^- server2 2 6 377 0 -4804us[-4804us] +/- 77ms
^* server3 2 6 377 59 -42us[ -46us] +/- 312us
^+ server4 3 6 377 60 +11ns[-3913ns] +/- 193us
Graphically, the transition is clear. Before the switch, offsets fluctuated within ±1.5 ms. After switching to chrony, offsets fell into the microsecond range and the estimated error window shrank to hundreds of microseconds. But these figures are still the daemon's own estimates — we needed independent verification.
Measuring real offset with 1PPS
To validate NTP accuracy, we can tap into the atomic clock's timing circuitry and extract a 1PPS signal — one pulse per second over a coaxial cable. Comparing the phase of that reference pulse against a pulse generated by the NTP server under test reveals the actual time delta. This requires specialized network interface cards with 1PPS support, which not all servers have.
Our first attempt was manual, using an oscilloscope to observe the phase shift. Within ten minutes of observation, we estimated ntpd's offset at roughly 3.5 ms, occasionally spiking to 10 ms.
That approach doesn't scale across a large fleet. A better method connects the test server's 1PPS output back into the Stratum 1 device's 1PPS input and monitors the difference continuously. This eliminates the need for an oscilloscope in a datacenter and provides ongoing, real-time measurements of NTP offset. The reference signal itself carries only nanoseconds of error, mostly from cable length.
These techniques have downsides that keep them out of routine production use:
- Cabling: Coaxial runs for 1PPS require datacenter design changes and are impractical for spot checks across sites.
- Specialized hardware: Network cards with 1PPS output are not standard.
- Infrastructure requirements: A Stratum 1 device with a 1PPS input is needed.
- Software overhead: The test server must run
ntpdto generate its pulse, adding user-space scheduling error to the measurement.
Validating the fixes with a lab-grade reference
To independently verify the accuracy of our NTP clients, we turned to purpose-built test devices. These units combine a GNSS receiver, an atomic clock, multiple 1PPS and network interfaces, and the ability to act as NTP clients themselves. This lets us run the same checks over the NTP protocol: the device records each received NTP packet with a timestamp taken directly from its atomic clock or GNSS receiver, bypassing the operating system's clock stack entirely.
This approach offers several advantages over our earlier 1PPS-based setup:
- No extra 1PPS cabling is needed. The atomic clock can be disciplined via GNSS or from a Stratum 1 device in the same rack with a short cable.
- Network packets are stamped using the atomic clock's data, making the OS's influence negligible — errors stay in the nanosecond range.
- The device supports both NTP and PTP (Precision Time Protocol).
- It is portable, allowing spot checks across different locations.
- Data can be exported to CSV, matching our internal data format.
ntpd: the 10 ms drop persists
These measurements closely match what we saw with daemon estimates and 1PPS measurements. A 10 ms drop appears immediately after startup and is slowly corrected to within ±1 ms. Strikingly, this drop is persistent and recurs after every daemon restart.
chrony: an order of magnitude better
Chrony's performance, measured by the lab device, is 10 to 100 times better than ntpd's, aligning with its own daemon estimates.
Hardware timestamps: squeezing out the last microseconds
Chrony's offset improves further when hardware timestamps are enabled. To understand why, we need to look at how NTP packets are handled. A client's NTP request contains a transmit timestamp field.
The server fills in fields like its receive timestamp, copies the client's transmit timestamp into the originate field, and returns the packet. The client then appends its own receive timestamp and calculates the offset using the formula from NTP RFC #958:
c = (t2 - t1 + t3 - t4)/2
This behavior can be observed with tcpdump:
08:37:31.489921 IP6 (hlim 127, next-header UDP (17) payload length: 56) client.36915 > time1.facebook.com.ntp: [bad udp cksum 0xf5d2 -> 0x595e!] NTPv4, length 48
Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecified), poll 3 (8s), precision -6
Root Delay: 1.000000, Root dispersion: 1.000000, Reference-ID: (unspec)
Reference Timestamp: 0.000000000
Originator Timestamp: 0.000000000
Receive Timestamp: 0.000000000
Transmit Timestamp: 3783170251.489887309 (2019/11/19 08:37:31)
Originator - Receive Timestamp: 0.000000000
Originator - Transmit Timestamp: 3783170251.489887309 (2019/11/19 08:37:31)
...
08:37:31.490923 IP6 (hlim 52, next-header UDP (17) payload length: 56) time1.facebook.com.ntp > server.36915: [udp sum ok] NTPv4, length 48
Server, Leap indicator: (0), Stratum 1 (primary reference), poll 3 (8s), precision -32
Root Delay: 0.000000, Root dispersion: 0.000152, Reference-ID: FB
Reference Timestamp: 3783169800.000000000 (2019/11/19 08:30:00)
Originator Timestamp: 3783170251.489887309 (2019/11/19 08:37:31)
Receive Timestamp: 3783170251.490613035 (2019/11/19 08:37:31)
Transmit Timestamp: 3783170251.490631213 (2019/11/19 08:37:31)
Originator - Receive Timestamp: +0.000725725
Originator - Transmit Timestamp: +0.000743903
...The problem is that Linux is not a real-time OS. Between the moment the transmit timestamp is written and the Write() call actually places data on the wire, there can be scheduler delays.
Even an ARP request, required when there's no existing entry for the destination, adds unaccounted time that inflates the error estimate. Chrony's hardware timestamps solve this by letting the network interface card (NIC) stamp the packet at the exact moment it's processed, with a residual delay of less than 10 ns before it physically leaves the port.
Referencing the chronyc sources output shown earlier:
^+ server4 2 6 377 60 +11ns[-3913ns] +/- 193usChrony reports an offset of 11 ns — the result of hardware timestamps enabled on both ends — though the estimated error remains in the hundreds of microseconds. Hardware timestamp support is increasingly common on modern NICs. To confirm your card has it, use ethtool to check for hardware-transmit and hardware-receive capabilities.
ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)
hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE)
hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)
Lab device measurements on the same hardware showed the real picture: ntpd ranged between -10 ms and 3 ms (a 13 ms spread), chrony between -200 µs and 200 µs, and chrony with hardware timestamps between -100 µs and 100 µs in most cases. This confirms that daemon estimates can be misleading.
Comparing against public services
Our internal tests were all conducted in controlled data-center environments. Running the same device against our public NTP endpoint and other well-known providers gives a different perspective:
These results depend heavily on the network path, connection speed, and quality. We ran the measurements multiple times from various locations over Wi-Fi and LAN networks unaffiliated with Facebook. Our public endpoint not only holds its own against popular providers; in several cases it outperforms them.
Designing the public NTP service
With internal precision at the sub-millisecond level, we launched a public NTP service. Clients can use time.facebook.com as their NTP server. To preserve user privacy, we do not fingerprint devices by IP address. The service runs out of our network PoPs (points of presence) across five geographically distributed endpoints:
- time1.facebook.com
- time2.facebook.com
- time3.facebook.com
- time4.facebook.com
- time5.facebook.com
Each endpoint terminates in a different location, improving both reliability against network path failures and time precision.
Network path for time2.facebook.com:
Network path for time3.facebook.com:
Handling leap seconds differently
Ntpd relies on the leap-seconds.list file, published in advance. This lets smearing start early so the time is already correct when the leap second actually occurs.
Chrony takes the opposite approach: it uses the leap-second indicator broadcast by GNSS hours ahead of the event. When the leap second happens at 00:00 UTC, chrony begins smearing it over a specified window. We chose chrony's methodology for our public service, smearing the leap second over approximately 18 hours after the event. Since this occurs simultaneously across many Stratum 2 servers, consistency matters. A smoother, sine-curve-based smear is also safer for applications than a sharp step.
What we learned
Measuring time is fundamentally challenging. Both ntpd and chrony produce estimates that are only partially accurate. For true monitoring of real offset, we recommend using 1PPS signals from a dedicated device or an external unit with a GNSS receiver and atomic clock.
Our comparison was decisive: chrony is significantly more precise than ntpd. This drove our migration of the entire infrastructure to chrony followed by the launch of the public NTP service. The effort paid off with an immediate leap in precision from tens of milliseconds down to hundreds of microseconds. Adding hardware timestamps improves precision by another two orders of magnitude.
Despite these gains, NTP has inherent limits. For teams needing even greater accuracy, evaluating PTP is the logical next step.
