Home/Meta Engineering
Source

Meta Engineering

381 articles from Meta Engineering.

SRE & Ops — Retrofitting null-safety onto Java at Meta

Retrofitting null-safety onto Java at Meta

We developed a new static analysis tool called Nullsafe that is used at Meta to detect NullPointerException (NPE) errors in Java code. Interoperability with legacy code and gradual deployment model were key to Nullsafe’s wide adoption and allowed us to recover some null-safety properties in the context of an otherwise null-unsafe language in a multimillion-line […]

CWChris Wiltz·November 22, 2022SRE & Ops
Networking — PTP: Timing accuracy and precision for the future of computing

PTP: Timing accuracy and precision for the future of computing

Meta is deploying a timing protocol, Precision Time Protocol (PTP), that will offer new levels of accuracy and precision to our networks and data centers. We believe PTP will become the global standard for keeping time in computer networks. PTP will benefit today’s products and services and will be a foundational technology behind the development […]

CWChris Wiltz·November 21, 2022Networking
Networking — How Precision Time Protocol is being deployed at Meta

How Precision Time Protocol is being deployed at Meta

Implementing Precision Time Protocol (PTP) at Meta allows us to synchronize the systems that drive our products and services down to nanosecond precision. PTP’s predecessor, Network Time Protocol (NTP), provided us with millisecond precision, but as we scale to more advanced systems on our way to building the next computing platform, the metaverse and AI, […]

CWChris Wiltz·November 21, 2022Networking
Career & Teams — Move faster, wait less: Improving code review time at Meta

Move faster, wait less: Improving code review time at Meta

Code reviews are one of the most important parts of the software development process At Meta we’ve recognized the need to make code reviews as fast as possible without sacrificing quality We’re sharing several tools and steps we’ve taken at Meta to reduce the time waiting for code reviews When done well, code reviews can […]

CWChris Wiltz·November 16, 2022Career & Teams
SRE & Ops — Sapling: Source control that’s user-friendly and scalable

Sapling: Source control that’s user-friendly and scalable

Sapling is a new Git-compatible source control client. Sapling emphasizes usability while also scaling to the largest repositories in the world. ReviewStack is a demonstration code review UI for GitHub pull requests that integrates with Sapling to make reviewing stacks of commits easy. You can get started using Sapling today. Source control is one of […]

CWChris Wiltz·November 15, 2022SRE & Ops
SRE & Ops — Tulip: Schematizing Meta’s data platform

Tulip: Schematizing Meta’s data platform

We’re sharing Tulip, a binary serialization protocol supporting schema evolution. Tulip assists with data schematization by addressing protocol reliability and other issues simultaneously. It replaces multiple legacy formats used in Meta’s data platform and has achieved significant performance and efficiency gains. There are numerous heterogeneous services, such as warehouse data storage and vario

CWChris Wiltz·November 9, 2022SRE & Ops
Networking — Reducing Instagram’s basic video compute time by 94 percent

Reducing Instagram’s basic video compute time by 94 percent

In our constant quest to prioritize efficiency, Instagram’s engineers have developed a way to process new videos that reduces the cost to produce basic video encodings by 94 percent. With this method in place, Meta’s video infrastructure can continue to scale without needing to add more machines. This frees up resources so more people can […]

CWChris Wiltz·November 4, 2022Networking
SRE & Ops — Improving Instagram notification management with machine learning and causal inference

Improving Instagram notification management with machine learning and causal inference

We’re sharing how Meta is applying statistics and machine learning (ML) to improve notification personalization and management on Instagram – particularly on daily digest push notifications. By using causal inference and ML to identify highly active users who are likely to see more content organically, we have been able to reduce the number of notifications […]

CWChris Wiltz·October 31, 2022SRE & Ops
Frontend — From zero to 10 million lines of Kotlin

From zero to 10 million lines of Kotlin

We’re sharing lessons learned from shifting our Android development from Java to Kotlin. Kotlin is a popular language for Android development and offers some key advantages over Java. As of today, our Android codebase contains over 10 million lines of Kotlin code. We’re open sourcing various examples and utilities we used to manipulate Kotlin code […]

CWChris Wiltz·October 24, 2022Frontend
SRE & Ops — OCP Summit 2022: Open hardware for AI infrastructure

OCP Summit 2022: Open hardware for AI infrastructure

At OCP Summit 2022, we’re announcing Grand Teton, our next-generation platform for AI at scale that we’ll contribute to the OCP community. We’re also sharing new innovations designed to support data centers as they advance to support new AI technologies: A new, more efficient version of Open Rack. Our Air-Assisted Liquid Cooling (AALC) – design. […]

CWChris Wiltz·October 18, 2022SRE & Ops
Frontend — Launching a new Chromium-based WebView for Android

Launching a new Chromium-based WebView for Android

Our in-app browser for Facebook on Android has historically relied on an Android System WebView based on Chromium, the open source project that powers many browsers on Android and other operating systems. On other mobile operating systems, the System WebView component cannot be updated without updating the entire operating system. On Android, this works differently, […]

CWChris Wiltz·September 30, 2022Frontend
SRE & Ops — Scaling data ingestion for machine learning training at Meta

Scaling data ingestion for machine learning training at Meta

Many of Meta’s products, such as search, ads ranking and Marketplace, utilize AI models to continuously improve user experiences. As the performance of hardware we use to support training infrastructure increases, we need to scale our data ingestion infrastructure accordingly to handle workloads more efficiently. GPUs, which are used for training infrastructure, tend to double […]

MEMeta Engineering·September 19, 2022SRE & Ops
SRE & Ops — How thermal simulation helps optimize Meta’s data centers

How thermal simulation helps optimize Meta’s data centers

Data center optimization has always played an important role at Meta. By optimizing our data centers’ environmental controls, we can reduce our environmental impact while ensuring that people can always depend on our products. With most other complex systems, optimization of energy consumption is a trial-and-error process. But experimenting on any component of a live […]

MEMeta Engineering·September 14, 2022SRE & Ops
SRE & Ops — MemLab: An open source framework for finding JavaScript memory leaks

MemLab: An open source framework for finding JavaScript memory leaks

We’ve open-sourced MemLab, a JavaScript memory testing framework that automates memory leak detection. Finding and addressing the root cause of memory leaks is important for delivering a quality user experience on web applications. MemLab has helped engineers and developers at Meta improve user experience and make significant improvements in memory optimization. We hope it will […]

CWChris Wiltz·September 12, 2022SRE & Ops
SRE & Ops — Open-sourcing TAOBench: An end-to-end social network benchmark

Open-sourcing TAOBench: An end-to-end social network benchmark

What the research is: The continued emergence of large social network applications has introduced a scale of data and query volume that challenges the limits of existing data stores. However, few benchmarks accurately simulate these request patterns, leaving researchers in short supply of tools to evaluate and improve upon these systems. To address this issue, […]

CWChris Wiltz·September 7, 2022SRE & Ops
Networking — Network Entitlement: A contract-based network sharing solution

Network Entitlement: A contract-based network sharing solution

Meta’s overall network usage and traffic volume has increased as we’ve continued to add new services. Due to the scarcity of fiber resources, we’re developing an explicit resource reservation framework to effectively plan, manage, and operate the shared consumption of network bandwidth, which will help us keep up with demand and limit network disruptions during […]

CWChris Wiltz·September 7, 2022Networking
SRE & Ops — Viewing the world as a computer: Global capacity management

Viewing the world as a computer: Global capacity management

Meta currently operates 14 data centers around the world. This rapidly expanding global data center footprint poses new challenges for service owners and for our infrastructure management systems. Systems like Twine, which we use to scale cluster management, and RAS, which handles perpetual region-wide resource allocation, have provided the abstractions and automation necessary for service […] Rea

CWChris Wiltz·September 6, 2022SRE & Ops
SRE & Ops — Improving Meta’s SLO workflows with data annotations

Improving Meta’s SLO workflows with data annotations

When we focus on minimizing errors and downtime here at Meta, we place a lot of attention on service-level indicators (SLIs) and service-level objectives (SLOs). Consider Instagram, for example. There, SLIs represent metrics from different product surfaces, like the volume of error response codes to certain endpoints, or the number of successful media uploads. Based […]

CWChris Wiltz·August 29, 2022SRE & Ops
Security — Five security principles for billions of messages across Meta’s apps

Five security principles for billions of messages across Meta’s apps

At Meta, our messaging apps help billions of people around the world stay connected to those who matter most to them. This scale brings potential threats from criminals and hackers, so we have a responsibility to keep people and their data safe. We’re sharing a set of principles to ensure that security is central to […]

CWChris Wiltz·July 28, 2022Security
SRE & Ops — Programming languages endorsed for server-side use at Meta

Programming languages endorsed for server-side use at Meta

Supporting a programming language at Meta is a very careful and deliberate decision. We’re sharing our internal programming language guidance that helps our engineers and developers choose the best language for their projects. Rust is the latest addition to Meta’s list of supported server-side languages. At Meta, we use many different programming languages for a […]

CWChris Wiltz·July 27, 2022SRE & Ops
Frontend — Launching Instagram Messaging on desktop

Launching Instagram Messaging on desktop

In 2020 we launched Instagram Messaging (referred to in this post simply as “Messaging”) for personal desktop computers. We believe that this feature will improve everyday experiences and enable new use cases for all of our desktop web users. In this post, we go through some of our overall learnings from our desktop users, and […]

CWChris Wiltz·July 26, 2022Frontend
SRE & Ops — It’s time to leave the leap second in the past

It’s time to leave the leap second in the past

The leap second concept was first introduced in 1972 by the International Earth Rotation and Reference Systems Service (IERS) in an attempt to periodically update Coordinated Universal Time (UTC) due to imprecise observed solar time (UT1) and the long-term slowdown in the Earth’s rotation. This periodic adjustment mainly benefits scientists and astronomers as it allows […]

CWChris Wiltz·July 25, 2022SRE & Ops
Security — Using Hermes’s Quicksort to run Doom: A tale of JavaScript exploitation

Using Hermes’s Quicksort to run Doom: A tale of JavaScript exploitation

At Meta, our Bug Bounty program is an important element of our “defense-in-depth” approach to security. Our internal product security teams investigate every bug submission to assess its maximum potential impact so that we can always reward external researchers based on both the bug they found and our further internal research assessment of where else […]

CWChris Wiltz·July 20, 2022Security
Frontend — Building text animations for Instagram Stories

Building text animations for Instagram Stories

In August 2020, Instagram launched a set of dynamic and fun text styles followed by animations to give people more choices to express themselves on Stories and Reels. This was the first major update to Stories’ text tools since 2016, and we wanted to share how we approached some obstacles we encountered and what we […]

MEMeta Engineering·July 18, 2022Frontend
SRE & Ops — Owl: Distributing content at Meta scale

Owl: Distributing content at Meta scale

Being able to distribute large, widely -consumed objects (so-called hot content) efficiently to hosts is becoming increasingly important within Meta’s private cloud. These are commonly distributed content types such as executables, code artifacts, AI models, and search indexes that help enable our software systems. Owl is a new system for high-fanout distribution of large data […]

CWChris Wiltz·July 14, 2022SRE & Ops
Networking — Watch Meta’s engineers discuss QUIC and TCP innovations for our network

Watch Meta’s engineers discuss QUIC and TCP innovations for our network

With more than 75 percent of our internet traffic set to use QUIC and HTTP/3 together, QUIC is slowly moving to become the de facto protocol used for internet communication at Meta. For Meta’s data center network, TCP remains the primary network transport protocol that supports thousands of services on top of it. As our […]

CWChris Wiltz·July 6, 2022Networking
SRE & Ops — Transparent memory offloading: more memory at a fraction of the cost and power

Transparent memory offloading: more memory at a fraction of the cost and power

Transparent memory offloading (TMO) is Meta’s data center solution for offering more memory at a fraction of the cost and power of existing technologies In production since 2021, TMO saves 20 percent to 32 percent of memory per server across millions of servers in our data center fleet We are witnessing massive growth in the […]

MEMeta Engineering·June 20, 2022SRE & Ops
AI & ML — Applying federated learning to protect data on mobile devices

Applying federated learning to protect data on mobile devices

What the research is: Federated learning with differential privacy (FL-DP) is one of the latest privacy-enhancing technologies being evaluated at Meta as we constantly work to enhance user privacy and further safeguard users’ data in the products we design, build, and maintain. FL-DP enhances privacy in two important ways: It allows machine learning (ML) models […]

CWChris Wiltz·June 14, 2022AI & ML
Frontend — Under the hood: Meta’s cloud gaming infrastructure

Under the hood: Meta’s cloud gaming infrastructure

The promise of cloud gaming is a promise to democratize gaming. Anyone who loves games should be able to enjoy them and share the experience with their friends, no matter where they’re located, and even if they don’t have the latest, most expensive gaming hardware. Facebook launched its cloud gaming platform in 2020 to give […]

CWChris Wiltz·June 9, 2022Frontend
SRE & Ops — Introducing Zelos: A ZooKeeper API leveraging Delos

Introducing Zelos: A ZooKeeper API leveraging Delos

Within large-scale services, durable storage, distributed leases, and coordination primitives such as distributed locks, semaphores, and events should be strongly consistent. At Meta, we have historically used Apache ZooKeeper as a centralized service for these primitives. However, as Meta’s workload has scaled, we’ve found ourselves pushing the limits of ZooKeeper’s capabilities. Modifying and tu

CWChris Wiltz·June 8, 2022SRE & Ops
SRE & Ops — Cache made consistent

Cache made consistent

Caches help reduce latency, scale read-heavy workloads, and save cost. They are literally everywhere. Caches run on your phone and in your browser. For example, CDNs and DNS are essentially geo-replicated caches. It’s thanks to many caches working behind the scenes that you can read this blog post right now. Phil Karlton famously said, “There […]

CWChris Wiltz·June 8, 2022SRE & Ops
SRE & Ops — Meta Open Source is transferring Jest to the OpenJS Foundation

Meta Open Source is transferring Jest to the OpenJS Foundation

Meta Open Source is officially transferring Jest, its open source JavaScript testing framework, to the OpenJS Foundation. With over 17 million weekly downloads and over 38,000 GitHub stars, Jest is the most used testing framework in the JavaScript ecosystem and is used by companies of all sizes, including Amazon, Google, Microsoft, and Stripe. We believe […]

CWChris Wiltz·May 11, 2022SRE & Ops
Frontend — Language packs: Meta’s mobile localization solution

Language packs: Meta’s mobile localization solution

More than 3 billion people around the world rely on our services each month. On mobile, around 57 percent of people on Facebook for Android and 49 percent of those on Facebook for iOS use the app in a language other than English. Delivering the best experience for these people, in their own language, is […]

MEMeta Engineering·May 9, 2022Frontend
SRE & Ops — BellJar: A new framework for testing system recoverability at scale

BellJar: A new framework for testing system recoverability at scale

Building infrastructure that can easily recover from outages, particularly outages involving adjacent infrastructure, too often becomes a murky exploration of nuanced fate-sharing between systems. Untangling dependencies and uncovering side effects of unavailability has historically been time-consuming work. A lack of great tooling built for this, and the rarity of infrastructure outages, makes re

CWChris Wiltz·May 5, 2022SRE & Ops
SRE & Ops — How the Cinder JIT’s function inliner helps us optimize Instagram

How the Cinder JIT’s function inliner helps us optimize Instagram

Since Instagram runs one of the world’s largest deployments of the Django web framework, we have natural interest in finding ways to optimize Python so we can speed up our production application. As part of this effort, we’ve recently open-sourced Cinder, our Python runtime that is a fork of CPython. Cinder includes optimizations like immortal […]

CWChris Wiltz·May 2, 2022SRE & Ops
Networking — A new way to survey potential fiber routes — without access to paved roads

A new way to survey potential fiber routes — without access to paved roads

Delivering on our vision for the metaverse will require us to reimagine a network infrastructure capable of supporting the computing platforms of the future. Although the metaverse is still a ways off, elements of it are already underway and we are already collaborating with telecommunications companies around the world to develop shared, open access optical […]

MEMeta Engineering·May 1, 2022Networking
SRE & Ops — How Meta enables de-identified authentication at scale

How Meta enables de-identified authentication at scale

Data minimization — collecting the minimum amount of data required to support our services — is one of our core principles at Meta as we continue developing new privacy-enhancing technologies (PETs). We are constantly seeking ways to improve privacy and protect user data on our family of products. Previously, we’ve approached data minimization by exploring […]

CWChris Wiltz·March 30, 2022SRE & Ops
SRE & Ops — Detecting silent errors in the wild: Combining two novel approaches to quickly detect silent data corruptions at scale

Detecting silent errors in the wild: Combining two novel approaches to quickly detect silent data corruptions at scale

Silent data corruptions (SDCs), data errors that go undetected by the larger system, are a widespread problem for large-scale infrastructure systems. Left undetected, these types of corruptions can cause data loss and propagate across the stack and manifest as application-level problems. Silent data corruptions (SDC) in hardware impact computational integrity for large-scale applications. Sources

CWChris Wiltz·March 17, 2022SRE & Ops
SRE & Ops — VESPA: Static profiling for binary optimization

VESPA: Static profiling for binary optimization

What the research is: Recent research has demonstrated that binary optimization is important for achieving peak performance for various applications. For instance, the state-of-the-art BOLT binary optimizer developed at Meta, which is part of the LLVM Compiler Project, significantly improves the performance of highly optimized binaries produced using compilers’ most aggressive optimizations, such

CWChris Wiltz·March 15, 2022SRE & Ops
Frontend — An open source compositional deadlock detector for Android Java

An open source compositional deadlock detector for Android Java

What the research is: We’ve developed a new static analyzer that catches deadlocks in Java code for Android without ever running the code. What distinguishes our analyzer from past research is its ability to analyze revisions in codebases with hundreds of millions of lines of code. We have deployed our analyzer in Meta’s continuous integration […]

CWChris Wiltz·March 8, 2022Frontend
SRE & Ops — Augmenting Flexible Paxos in LogDevice to improve read availability

Augmenting Flexible Paxos in LogDevice to improve read availability

We’ve improved read availability in LogDevice, Meta’s scalable distributed log storage system, by removing a fundamental trade-off in Flexible Paxos, the algorithm used to gain consensus among our distributed systems. At Meta’s scale, systems need to be reliable, even in the face of organic failures like power loss events, or when systems are undergoing hardware […]

CWChris Wiltz·March 7, 2022SRE & Ops
SRE & Ops — FOQS: Making a distributed priority queue disaster-ready

FOQS: Making a distributed priority queue disaster-ready

Facebook Ordered Queueing Service (FOQS) is a fully managed, distributed priority queueing service used for reliable message delivery among many services. FOQS has evolved from a regional deployment into a geo-distributed, global deployment to help ensure that data stored within logical queues is highly available, even through large-scale disaster scenarios. Migrating to a global architecture

MEMeta Engineering·January 18, 2022SRE & Ops
SRE & Ops — Power Loss Siren: Making Meta resilient to power loss events

Power Loss Siren: Making Meta resilient to power loss events

There are thousands of distributed services running on millions of servers in Meta’s data centers. Part of ensuring the reliability of those services means making them resilient to power loss events as our data center fleet grows. To help increase resiliency, we built the Power Loss Siren (PLS) — a rack level, low latency, distributed […]

CWChris Wiltz·December 16, 2021SRE & Ops
Security — Charting the future of our bug bounty program

Charting the future of our bug bounty program

We’re tackling the industry-wide issue of scraping by expanding our bug bounty program to reward valid reports of scraping bugs and unprotected data sets. To the best of our knowledge, this is an industry first. Looking toward the future, we’re also launching new educational opportunities for researchers and hosting our first BountyConEDU — a three-day […]

MEMeta Engineering·December 15, 2021Security
SRE & Ops — SLICK: Adopting SLOs for improved reliability

SLICK: Adopting SLOs for improved reliability

We would like to thank Peter Tang for all his work on SLICK, and for helping us write this post! To support the people and communities who use our apps and products, we need to stay in constant contact with them. We want to provide the experiences we offer reliably. We also need to establish […]

MEMeta Engineering·December 13, 2021SRE & Ops
SRE & Ops — OCP Summit 2021: Open networking hardware lays the groundwork for the metaverse

OCP Summit 2021: Open networking hardware lays the groundwork for the metaverse

Open infrastructure technologies and networking hardware will play an important role as we build new technologies for the metaverse, where billions of people will someday come together in virtual spaces. As we head toward the next major computing platform with a continued spirit of embracing openness and disaggregation, we’re announcing two new milestones for our […]

CWChris Wiltz·November 9, 2021SRE & Ops
SRE & Ops — Kangaroo: A new flash cache optimized for tiny objects

Kangaroo: A new flash cache optimized for tiny objects

What the research is: Kangaroo is a new flash cache that enables more efficient caching of tiny objects (objects that are ~100 bytes or less) and overcomes the challenges presented by existing flash cache designs. Since Kangaroo is implemented within CacheLib, Facebook’s open source caching engine, developers can use Kangaroo through CacheLib’s API to build […]

CWChris Wiltz·October 26, 2021SRE & Ops
SRE & Ops — Autonomous testing of services at scale

Autonomous testing of services at scale

Enabling developers to prototype, test, and iterate on new features quickly is important to Facebook’s success. To do this effectively, it’s key to have a stable infrastructure that doesn’t introduce unnecessary friction. This gets significantly more challenging when the infrastructure in question must also scale to support more than 3 billion people around the world, […]

MEMeta Engineering·October 20, 2021SRE & Ops
Security — Facebook engineers receive 2021 IEEE Computer Society Cybersecurity Award for static analysis tools

Facebook engineers receive 2021 IEEE Computer Society Cybersecurity Award for static analysis tools

Until recently, static analysis tools weren’t seen by our industry as a reliable element of securing code at scale. After nearly a decade of investing in refining these systems, I’m so proud to celebrate our engineering teams today for being awarded the IEEE Computer Society’s Cybersecurity Award for Practice for development and deployment of static […]

MEMeta Engineering·October 20, 2021Security
Networking — RTMP Go Away: Lossless reconnections for live streaming

RTMP Go Away: Lossless reconnections for live streaming

What it is: Real Time Messaging Protocol (RTMP) is a popular media streaming protocol that uses Transmission Control Protocol (TCP) persistent connections. When a connection between a live-streaming client and the platform is interrupted, data from the live event is lost until the client can reconnect to a new server. RTMP Go Away is a […]

CWChris Wiltz·October 19, 2021Networking
Networking — More details about the October 4 outage

More details about the October 4 outage

Now that our platforms are up and running as usual after yesterday’s outage, I thought it would be worth sharing a little more detail on what happened and why — and most importantly, how we’re learning from it. This outage was triggered by the system that manages our global backbone network capacity. The backbone is […]

MEMeta Engineering·October 5, 2021Networking
Networking — Update about the October 4th outage

Update about the October 4th outage

To all the people and businesses around the world who depend on us, we are sorry for the inconvenience caused by today’s outage across our platforms. We’ve been working as hard as we can to restore access, and our systems are now back up and running. The underlying cause of this outage also impacted many […]

MEMeta Engineering·October 4, 2021Networking
Frontend — Open-sourcing Mariana Trench: Analyzing Android and Java app security in depth

Open-sourcing Mariana Trench: Analyzing Android and Java app security in depth

We’re sharing details about Mariana Trench (MT), a tool we use to spot and prevent security and privacy bugs in Android and Java applications. As part of our effort to help scale security through building automation, we recently open-sourced MT to support security engineers at Facebook and across the industry. This post is the third […]

CWChris Wiltz·September 29, 2021Frontend
Networking — 2Africa Pearls subsea cable connects Africa, Europe, and Asia to bring affordable, high-speed internet to 3 billion people

2Africa Pearls subsea cable connects Africa, Europe, and Asia to bring affordable, high-speed internet to 3 billion people

Facebook invests in subsea cables as part of our continued efforts to build the infrastructure that carries internet traffic and helps bring more people online to a faster internet. Today, we are thrilled to announce, along with regional and global partners, a new segment of subsea cable called 2Africa Pearls, which connects three continents — […]

MEMeta Engineering·September 28, 2021Networking
Frontend — Superpack: Pushing the limits of compression in Facebook’s mobile apps

Superpack: Pushing the limits of compression in Facebook’s mobile apps

Managing app size at Facebook is a unique challenge: Every day, developers check in large volumes of code, and each line of code translates into additional bits in the apps that people ultimately download onto their phones. Left unchecked, this added code would make the app bigger and bigger until eventually the time it takes […]

MEMeta Engineering·September 13, 2021Frontend