
For years, in order to safeguard the privacy of people’s messages, WhatsApp has provided end-to-end encryption by default so messages can be seen only by the sender and recipient, and no one in between. Now, we’re planning to give people the option to protect their WhatsApp backups using end-to-end encryption as well. People can already […]
CWChris Wiltz·September 10, 2021Security 
Caching plays an important role in helping people access their information efficiently. For example, when an email app loads, it temporarily caches some messages, so the user can refresh the page without the app retrieving the same messages. However, large-scale caching has long been a complex engineering challenge. Companies must balance the fast experience people […]
MEMeta Engineering·September 2, 2021SRE & Ops 
What the research is: RAMP-TAO is a new protocol that improves the developer experience on TAO, Facebook’s online social graph store, by providing stronger transactional guarantees. It is the first protocol to provide transactional semantics over an eventually consistent massive-scale data store while still preserving the system’s overall reliability and performance. RAMP-TAO enables an intuitive

We are excited to announce our participation in the Apricot subsea cable system, together with leading regional and global partners. When completed, the project (which is still subject to regulatory approvals) will deliver much-needed internet capacity, redundancy, and reliability to expand connections in the Asia-Pacific region. The 12,000-kilometer-long cable will connect Japan, Taiwan, Guam, th

Facebook engineers have built and open-sourced an Open Compute Time Appliance, an important component of the modern timing infrastructure. To make this possible, we came up with the Time Card — a PCI Express (PCIe) card that can turn almost any commodity server into a time appliance. With the help of the OCP community, we […]
MEMeta Engineering·August 11, 2021Frontend 
What the research is: A first-of-its-kind study detailing our backbone management strategy to ensure high service performance throughout the COVID-19 pandemic. The pandemic moved most social interactions online and caused an unprecedented stress test on our global network infrastructure with tens of data center regions. At this scale, failures such as fiber cuts, router misconfigurations, […] Read

ZippyDB is the largest strongly consistent, geographically distributed key-value store at Facebook. Since we first deployed ZippyDB in 2013, this key-value store has expanded rapidly, and today, ZippyDB serves a number of use cases, ranging from metadata for a distributed filesystem, counting events for both internal and external purposes, to product data that’s used for […]

We are releasing Winterfell, our implementation of a STARK prover/verifier to Crates.io Winterfell is an easy to use open source implementation of STARKs for security and privacy applications. One potential application for Winterfell’s zero-knowledge proofs is blockchain privacy and scalability. “Any sufficiently advanced technology is indistinguishable from magic.” —Clarke’s Third Law What if the
MEMeta Engineering·August 4, 2021Security 
Whether it’s iterating on Facebook’s News Feed ranking algorithm or delivering the most relevant ads to users, we are constantly exploring new features to help improve our machine learning (ML) models. Every time we add new features, we create a challenging data engineering problem that requires us to think strategically about the choices we make. […]

MySQL, an open source database developed by Oracle, powers some of Facebook’s most important workloads. We actively develop new features in MySQL to support our evolving requirements. These features change many different areas of MySQL, including client connectors, storage engine, optimizer, and replication. Each new major version of MySQL requires significant time and effort to […]

Training AI models at a large scale isn’t easy. Aside from the need for large amounts of computing power and resources, there is also considerable engineering complexity behind training very large models. At Facebook AI Research (FAIR) Engineering, we have been working on building tools and infrastructure to make training large AI models easier. Our […]

For years, people have been asking us to create a true multi-device experience that allows people to use WhatsApp on other devices without requiring a smartphone connection. Today, we’re announcing the rollout of a limited public beta test for WhatsApp’s updated multi-device capability. With this new capability, you can now use WhatsApp on your phone […]

Our infrastructure supports thousands of services that handle billions of requests per second. We’ve previously discussed how we built our service encryption infrastructure to keep these globally distributed services operating securely and performantly. This post discusses the system we designed to enforce encryption policies within our network and shares some of the lessons we learned […] Read Mo
MEMeta Engineering·July 12, 2021Security 
What the research is: The Ribbon filter is a new data structure that is more space-efficient than the popular Bloom filters that are widely used for optimizing data retrieval. One of the ways that Bloom, and now Ribbon, filters solve real engineering problems is by providing smooth configurability unmatched by other filters. Bloom filters work […]

Accelerators are special-purpose hardware devices optimized for specific applications, like AI prediction and video encoding. And Application-specific hardware platforms play an important role in meeting the growing latency and compute demands of workloads like deep learning, content understanding, and video encoding. At Facebook, the inevitable rise in use of accelerators in our data centers has

Connectivity is an integral part of Facebook’s mission to bring people closer together, and the COVID-19 pandemic has only heightened the demand for critical internet access. According to the latest edition of our Inclusive Internet Index, nearly 70 percent of people around the world believe that increased internet usage in all aspects of their lives […]

What the research is: Tectonic, our data center scale distributed file system, enables better resource utilization, promotes simpler services, and requires less operational complexity than our previous approach. Our previous storage infrastructure consisted of a set of use-case specific storage systems. Clusters, or instances of these storage systems, used to scale to tens of petabytes. […] Read M

What it is: A new library to analyze time series data. Kats is a lightweight, easy-to-use, and generalizable framework for generic time series analysis, including forecasting, anomaly detection, multivariate analysis, and feature extraction/embedding. To the best of our knowledge, Kats is the first comprehensive Python library for generic time series analysis, which provides both classical […] Rea
MEMeta Engineering·June 21, 2021AI & ML 
Our production backbone network connects our data centers and delivers content to our users. The network supports a vast number of different services, distributed across a multitude of data centers. Traffic patterns shift over time from one data center to another due to the introduction of new services, service architecture changes, changes in user behavior, […]

Peripheral component interconnect express (PCIe) hardware continues to push the boundaries of computing thanks to advances in transfer speeds, the number of available lanes for simultaneous data delivery, and a comparatively small footprint on motherboards. Today, PCIe connectivity-based hardware delivers faster data transfers and is one of the de facto methods to connect components to […] Read Mo

Traffic on the internet travels across many different kinds of links. A fast and reliable way to exchange traffic between different networks and service providers is through peering. Initially, we managed peering via a time-intensive manual process. Reliable peering is essential for Facebook and for everyone’s internet use. But there is no industry standard for […]

What the research is: A first-of-its-kind study that details the scalable design, software implementation, and operations of Facebook’s data center routing design, based on Border Gateway Protocol (BGP). BGP was originally designed to interconnect autonomous internet service providers (ISPs) on the global internet. Highly scalable and widely acknowledged as an attractive choice for routing, BGP […

Facebook is embracing Rust, one of the most loved and fastest-growing programming languages available today. In addition to bringing new talent to its Rust team, Facebook has announced that it is officially joining the nonprofit Rust Foundation. Alongside fellow members including Mozilla (the creators of Rust), AWS, Microsoft, and Google, Facebook will be working to […]

Say you receive an email notification that a service is crashing just after your last code change deploys. The crash happens in only 0.1 percent of the servers where it runs. But you’re at a large-scale company, so 0.1 percent equals thousands of servers — and this issue is going to be hard to reproduce. Several […]

At WhatsApp, privacy is our DNA. That’s why we rolled out end-to-end encryption in 2016 — so that when messages are end-to-end encrypted, only you and your intended recipients can see the messages you send. But securing messages and calls is just one part of how we minimize the information we collect in the process […]

People upload hundreds of millions of videos to Facebook every day. Making sure every video is delivered at the best quality — with the highest resolution and as little buffering as possible — means optimizing not only when and how our video codecs compress and decompress videos for viewing, but also which codecs are used […]

We are announcing two vital new subsea cables to connect Singapore, Indonesia, and North America. These will be the first transpacific cables through a new diverse route crossing the Java Sea and will increase overall transpacific capacity by 70 percent. We are committed to bringing more people online to a faster internet. As part of […]

In 2019, Instagram launched a new app for direct messaging called Threads. We built media sends in Threads using a lot of the existing infrastructure at Instagram. Many of these components do not focus on the send latency. For this post, we’ll define “send latency” as the time it takes from when the user starts a […]

What the research is: Jump-Start is a new approach for improving the performance of virtual machines at scale. Virtual machines are a modern and popular design to implement programming languages used to build applications in general, including large-scale websites like Facebook and Instagram. However, virtual machines incur well-known performance overhead in terms of the amount […]

What the research is: Silent data corruption, or data errors that go undetected by the larger system, is a widespread problem for large-scale infrastructure systems. This type of corruption can propagate across the stack and manifest as application-level problems. It can also result in data loss and require months to debug and resolve. This work […]
MEMeta Engineering·February 23, 2021SRE & Ops 
We will be hosting a talk about our work on Scaling a Distributed Priority Queue during our virtual Systems @Scale event at 11 am PT on Wednesday, February 24, followed by a live Q&A session. Please submit any questions to [email protected] before the event. The entire Facebook ecosystem is powered by thousands of distributed systems […]

Every workday, Facebook engineers commit thousands of diffs (which is a change consisting of one or more files) into production. This code velocity allows us to rapidly ship new features, deliver bug fixes and optimizations, and run experiments. However, a natural downside to moving quickly in any industry is the risk of inadvertently causing regressions […]
MEMeta Engineering·February 17, 2021SRE & Ops 
Root cause analysis (RCA) is an important part of fixing any bug. After all, you can’t solve a problem without getting to the heart of it. But RCA isn’t always simple, especially at a scale like Facebook’s. When billions of people are using an app on a variety of platforms and devices, a single bug […]

What it is: Thrift is a serialization and remote procedure call (RPC) framework used for cross-service communication. Most services at Facebook communicate via Thrift because it provides a simple, language-agnostic protocol for communicating with structured data. Thrift can already be used in programming languages such as C++, Python, and Java using fbthrift. We are also […]

Designing a personalized ranking system for more than 2 billion people (all with different interests) and a plethora of content to select from presents significant, complex challenges. This is something we tackle every day with News Feed ranking. Without machine learning (ML), people’s News Feeds could be flooded with content they don’t find as relevant […]

It goes without saying that 2020 has been a challenging year, to put it lightly. But if anything, the COVID-19 pandemic has shined a light on our need to connect as people. For Facebook, that meant our work has become more important than ever. Whether it was finding new and innovative ways to expand internet […]

We are rolling out a new video calling library to all the relevant products across our apps and services, including Instagram, Messenger, Portal, Workplace chat, etc. To create a library generic enough to support all these different use cases, we needed to rewrite our existing library from scratch using the latest version of the open […]

Together with mobile network operators Airtel and Vodafone, and technology partners ADI, Altiostar, Baicells, Flex, Mavenir, MTI, Parallel Wireless, and Xilinx, we are focused on building general-purpose radio access network (RAN) reference designs for 4G and 5G networks in the Open RAN ecosystem. This work is being done as part of the Evenstar program. The […]

One of the biggest questions faced by the recommendation systems community is whether recommendation systems should be solely evaluated by their predictive accuracy, or also focus on other factors such as preference broadening. In order to appreciate the importance of this question, let us look at a couple of simulated scenarios in a dynamic setting. User […]
CWChris Wiltz·December 17, 2020AI & ML 
In the spring of 2019 — a full year before the COVID-19 pandemic caused the entire world to turn our living rooms into offices, schools, and gathering places — Mark Zuckerberg shared a plan to create the digital equivalent of the living room, where people could connect and hang out together. That work was already underway early […]
MEMeta Engineering·December 16, 2020Frontend 
What it is: Infer# brings the Infer static analysis platform to developers who use Microsoft’s C# programming language. It can already detect null-pointer dereference and resource leak bugs, thanks to bi-abduction analysis. Detection of race conditions based on RacerD analysis is also in the works. Infer# has been used to analyze Microsoft software, including Roslyn, […]

Designing a constrained exploration system […]

Facebook’s codebase changes each day as engineers develop new features and optimizations for our apps. If not validated, each of these changes could potentially regress the functionality or reliability of our products for billions of people around the world. To mitigate this risk, we maintain an enormous suite of automated regression tests to cover various […]
MEMeta Engineering·December 10, 2020SRE & Ops 
Facebook’s services rely on fleets of servers in data centers all over the globe — all running applications and delivering the performance our services need. This is why we need to make sure our server hardware is reliable and that we can manage server hardware failures at our scale with as little disruption to our […]

As part of our efforts to help expand connectivity around the world, Facebook Connectivity has been prototyping SuperCell, a wide-area coverage solution for increasing mobile connectivity in rural communities. Now, after working with telecom industry partners to conduct several trials and data analyses, we’re ready to share what we’ve learned. Rather than produce this solution […]

High-quality internet connectivity gives people a voice, creates opportunities, and strengthens local and global economies. The need for widespread reliable internet connectivity and infrastructure is more apparent now than ever, as heavier reliance on remote work and online communication during the COVID-19 pandemic drives a dramatic surge in global internet usage. And yet, according to […] Read

What it is: FioSynth is a benchmark tool used to automate the execution of storage workload suites and to parse results. It contains a base set of block level storage workloads, synthesized from production I/O traces, that simulate a diverse range of Facebook production services. It is useful for predicting how a storage device will […]
MEMeta Engineering·November 18, 2020SRE & Ops 
What the research is: Twine is our homegrown cluster management system, which has been running in production for the past decade. A cluster management system allocates workloads to machines and manages the life cycle of machines, containers, and workloads. Kubernetes is a prominent example of an open source cluster management system. Twine has helped convert […]
MEMeta Engineering·November 11, 2020SRE & Ops 
As Facebook Live and Facebook Watch have grown, the content shared there has expanded to include professionally produced, broadcast-quality content, like shows and live event coverage. This type of professional broadcasting from providers like La Liga, CONMEBOL, and UEFA is a very different challenge from user generated content (UGC) and viewers’ expectations are much higher. […]

We are replacing the de facto protocol the internet has used for decades with QUIC, the latest and most radical step we’ve taken to optimize our network protocols to create a better experience for people on our services. Today, more than 75 percent of our internet traffic uses QUIC and HTTP/3 (we refer to QUIC […]

What it is: We all want our workloads to run on our machines as efficiently as possible. With the Facebook resource control demo (resctl-demo), developers can simulate system resource conflicts and test ways to resolve them. It’s like going on a guided tour through a system’s resource control, complete with live demos and detailed explanations. […]

Large-scale companies serve millions or even billions of people who depend on the services these companies provide for their everyday needs. To keep these services running and delivering meaningful experiences, the teams behind them need to find the most relevant and accurate information quickly so that they can make informed decisions and take action. Finding […]

What it is: CG/SQL is a code generation system for the popular SQLite library that allows developers to write stored procedures in a variant of Transact-SQL (T-SQL) and compile them into C code that uses SQLite’s C API to do the coded operations. CG/SQL enables engineers to create highly complex stored procedures with very large […]

The U.K.’s Royal Academy of Engineering has elected Facebook Research Scientist Mark Harman as a Fellow for his achievements in academia and industry, including his work on search-based software engineering (SBSE), intelligent software testing tools, and web-enabled simulation (WES) approaches. Election to the Academy is by invitation only, and it is one of the highest […]
MEMeta Engineering·September 21, 2020SRE & Ops 
In 2018, we set an ambitious goal of achieving a 75 percent absolute reduction in operational greenhouse gas (GHG) emissions and to support our global operations with 100 percent renewable energy by the end of 2020. On both counts, we are well on track: These commitments have spurred the construction of over 5,400 megawatts (MW) […]
MEMeta Engineering·September 14, 2020SRE & Ops 
Facebook’s web tier is one of the main services that handle HTTP requests from people using our services each time they interact with Facebook. It is a massive global service distributed across multiple data centers throughout the world. Since we have people from all over the world using our services, the load on the web […]

As our infrastructure has expanded, we have seen an exponential growth in failures that affect a subset of capacity in a data center. These may stem from software or firmware errors, or issues in the mechanical or electrical equipment in the data center. As our data center footprint grows, so does the frequency of such […]

The Jupiter subsea cable is a transpacific cable that runs between Japan and the Philippines and the U.S., where it lands in Hermosa Beach, California, and Tierra del Mar, Oregon. During Facebook’s Oregon landing drilling operation in Tierra del Mar on April 28, an unfortunate incident occurred, in which a horizontal directional drill hit hard […]

Hardware fails, networks partition, and humans break things. The job of our infrastructure engineers is to abstract these realities away and provide a reliable, stable production environment nonetheless. Two of the technologies we deploy in this pursuit are Twine, our internal container orchestrator, and Apache ZooKeeper. Twine maintains service availability by managing services in containers

Over the years, as we’ve expanded in scale and functionalities, Facebook has evolved from a basic web server architecture into a complex one with thousands of services working behind the scenes. It’s no trivial task to scale the wide range of back-end services needed for Facebook’s products. And we found that many of our teams […]