Home/Meta Engineering
Source

Meta Engineering

381 articles from Meta Engineering.

AI & ML — Optimizing RTC bandwidth estimation with machine learning

Optimizing RTC bandwidth estimation with machine learning

Bandwidth estimation (BWE) and congestion control play an important role in delivering high-quality real-time communication (RTC) across Meta’s family of apps. We’ve adopted a machine learning (ML)-based approach that allows us to solve networking problems holistically across cross-layers such as BWE, network resiliency, and transport. We’re sharing our experiment results from this approach, some

CWChris Wiltz·March 20, 2024AI & ML
Engineering — Better video for mobile RTC with AV1 and HD

Better video for mobile RTC with AV1 and HD

At Meta, we support real-time communication (RTC) for billions of people through our apps, including Messenger, Instagram, and WhatsApp. We’ve seen significant benefits by adopting the AV1 codec for RTC. Here’s how we are improving the RTC video quality for our apps with tools like the AV1 codec, the challenges we face, and how we […]

CWChris Wiltz·March 20, 2024Engineering
SRE & Ops — Logarithm: A logging engine for AI training workflows and services

Logarithm: A logging engine for AI training workflows and services

Systems and application logs play a key role in operations, observability, and debugging workflows at Meta. Logarithm is a hosted, serverless, multitenant service, used only internally at Meta, that consumes and indexes these logs and provides an interactive query interface to retrieve and view logs. In this post, we present the design behind Logarithm, and […]

CWChris Wiltz·March 18, 2024SRE & Ops
AI & ML — Building Meta’s GenAI Infrastructure

Building Meta’s GenAI Infrastructure

Marking a major investment in Meta’s AI future, we are announcing two 24k GPU clusters. We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training. We are strongly committed to open […]

CWChris Wiltz·March 12, 2024AI & ML
Security — Making messaging interoperability with third parties safe for users in Europe

Making messaging interoperability with third parties safe for users in Europe

To comply with a new EU law, the Digital Markets Act (DMA), which comes into force on March 7th, we’ve made major changes to WhatsApp and Messenger to enable interoperability with third-party messaging services. We’re sharing how we enabled third-party interoperability (interop) while maintaining end-to-end encryption (E2EE) and other privacy guarantees in our services as […]

CWChris Wiltz·March 6, 2024Security
Career & Teams — How DotSlash makes executable deployment simpler

How DotSlash makes executable deployment simpler

Andres Suarez and Michael Bolin, two software engineers at Meta, join Pascal Hartig (@passy) on the Meta Tech Podcast to discuss the ins and outs of DotSlash, a new open source tool from Meta. DotSlash takes the pain out of distributing binaries and toolchains to developers. Instead of committing large, platform-specific executables to a repository, […]

CWChris Wiltz·February 26, 2024Career & Teams
SRE & Ops — Aligning Velox and Apache Arrow: Towards composable data management

Aligning Velox and Apache Arrow: Towards composable data management

We’ve partnered with Voltron Data and the Arrow community to align and converge Apache Arrow with Velox, Meta’s open source execution engine. Apache Arrow 15 includes three new format layouts developed through this partnership: StringView, ListView, and Run-End-Encoding (REE). This new convergence helps Meta and the larger community build data management systems that are unified, […]

CWChris Wiltz·February 20, 2024SRE & Ops
Career & Teams — Meta loves Python

Meta loves Python

By now you’re already aware that Python 3.12 has been released. But did you know that several of its new features were developed by Meta? Meta engineer Pascal Hartig (@passy) is joined on the Meta Tech Podcast by Itamar Oren and Carl Meyer, two software engineers at Meta, to discuss their teams’ contributions to the […]

CWChris Wiltz·February 12, 2024Career & Teams
Networking — Simple Precision Time Protocol at Meta

Simple Precision Time Protocol at Meta

While deploying Precision Time Protocol (PTP) at Meta, we’ve developed a simplified version of the protocol (Simple Precision Time Protocol – SPTP), that can offer the same level of clock synchronization as unicast PTPv2 more reliably and with fewer resources. In our own tests, SPTP boasts comparable performance to PTP, but with significant improvements in […]

CWChris Wiltz·February 7, 2024Networking
SRE & Ops — DotSlash: Simplified executable deployment

DotSlash: Simplified executable deployment

We’ve open sourced DotSlash, a tool that makes large executables available in source control with a negligible impact on repository size, thus avoiding I/O-heavy clone operations. With DotSlash, a set of platform-specific executables is replaced with a single script containing descriptors for the supported platforms. DotSlash handles transparently fetching, decompressing, and verifying the appropr

CWChris Wiltz·February 6, 2024SRE & Ops
AI & ML — Improving machine learning iteration speed with faster application build and packaging

Improving machine learning iteration speed with faster application build and packaging

Slow build times and inefficiencies in packaging and distributing execution files were costing our ML/AI engineers a significant amount of time while working on our training stack. By addressing these issues head-on, we were able to reduce this overhead by double-digit percentages. In the fast-paced world of AI/ML development, it’s crucial to ensure that our […]

CWChris Wiltz·January 29, 2024AI & ML
SRE & Ops — Lazy is the new fast: How Lazy Imports and Cinder accelerate machine learning at Meta

Lazy is the new fast: How Lazy Imports and Cinder accelerate machine learning at Meta

At Meta, the quest for faster model training has yielded an exciting milestone: the adoption of Lazy Imports and the Python Cinder runtime. The outcome? Up to 40 percent time to first batch (TTFB) improvements, along with a 20 percent reduction in Jupyter kernel startup times. This advancement facilitates swifter experimentation capabilities and elevates the […]

CWChris Wiltz·January 18, 2024SRE & Ops
AI & ML — How Meta is advancing GenAI

How Meta is advancing GenAI

What’s going on with generative AI (GenAI) at Meta? And what does the future have in store? In this episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) speaks with Devi Parikh, an AI research director at Meta. They cover a wide range of topics, including the history and future of GenAI and the most […]

CWChris Wiltz·January 11, 2024AI & ML
SRE & Ops — How Meta built the infrastructure for Threads

How Meta built the infrastructure for Threads

On July 5, 2023, Meta launched Threads, the newest product in our family of apps, to an unprecedented success that saw it garner over 100 million sign ups in its first five days. A small, nimble team of engineers built Threads over the course of only five months of technical work. While the app’s production […]

CWChris Wiltz·December 19, 2023SRE & Ops
SRE & Ops — AI debugging at Meta with HawkEye

AI debugging at Meta with HawkEye

HawkEye is the powerful toolkit used internally at Meta for monitoring, observability, and debuggability of the end-to-end machine learning (ML) workflow that powers ML-based products. HawkEye supports recommendation and ranking models across several products at Meta. Over the past two years, it has facilitated order of magnitude improvements in the time spent debugging production issues. […] Read

CWChris Wiltz·December 19, 2023SRE & Ops
Security — Building end-to-end security for Messenger

Building end-to-end security for Messenger

We are beginning to upgrade people’s personal conversations on Messenger to use end-to-end encryption (E2EE) by default. Meta is publishing two technical white papers on end-to-end encryption: Our Messenger end-to-end encryption whitepaper describes the core cryptographic protocol for transmitting messages between clients. The Labyrinth encrypted storage protocol whitepaper explains our protocol f

CWChris Wiltz·December 6, 2023Security
SRE & Ops — Writing and linting Python at scale

Writing and linting Python at scale

Python plays a big part at Meta. It powers Instagram’s backend and plays an important role in our configuration systems, as well as much of our AI work. Meta even made contributions to Python 3.12, the latest version of Python. On this episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) is joined by Amethyst […]

CWChris Wiltz·November 21, 2023SRE & Ops
AI & ML — Watch: Meta’s engineers on building network infrastructure for AI

Watch: Meta’s engineers on building network infrastructure for AI

Meta is building for the future of AI at every level – from hardware like MTIA v1, Meta’s first-generation AI inference accelerator to publicly released models like Llama 2, Meta’s next-generation large language model, as well as new generative AI (GenAI) tools like Code Llama. Delivering next-generation AI products and services at Meta’s scale also […]

CWChris Wiltz·November 15, 2023AI & ML
Security — Enhancing the security of WhatsApp calls

Enhancing the security of WhatsApp calls

New optional features in WhatsApp have helped make calling on WhatsApp more secure. “Silence Unknown Callers” is a new setting on WhatsApp that not only quiets annoying calls but also blocks sophisticated cyber attacks. “Protect IP Address in Calls” is a new setting on WhatsApp that helps hide your location from other parties on the […]

CWChris Wiltz·November 8, 2023Security
Frontend — How Meta built Threads in 5 months

How Meta built Threads in 5 months

In about five short months, a small team of engineers at Meta took Threads, the new text-based conversations app, from from an idea to the most successful app launch of all time, pulling in over 100M users in its first five days. But this achievement wouldn’t have been possible without Meta’s existing systems and infrastructure. […]

CWChris Wiltz·November 6, 2023Frontend
SRE & Ops — Automating data removal

Automating data removal

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. SCARF scans production data systems to identify tables or assets that are unused and safely removes them. SCARF avoids tedious manual work and ensures that product data is correctly removed when a product is shut down. This […]

CWChris Wiltz·October 31, 2023SRE & Ops
SRE & Ops — Automating dead code cleanup

Automating dead code cleanup

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing dead code. SCARF combines static and dynamic analysis of programs to detect dead code from both a business and programming language perspective. SCARF automatically creates change requests that delete the dead code identified from the program analysis, minimizing developer costs. […]

CWChris Wiltz·October 24, 2023SRE & Ops
SRE & Ops — 5 Things you didn’t know about Buck2

5 Things you didn’t know about Buck2

Meta has a very large monorepo, with many different programming languages. To optimize build and performance, we developed our own build system called Buck, which was first open-sourced in 2013. Buck2 is the recently open-sourced successor. In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x as fast as Buck1. Below […]

CWChris Wiltz·October 23, 2023SRE & Ops
SRE & Ops — Automating product deprecation

Automating product deprecation

Systematic Code and Asset Removal Framework (SCARF) is Meta’s unused code and data deletion framework. SCARF guides engineers through deprecating a product safely and efficiently via an internal tool. SCARF combines this tooling with automation to reduce load on engineers. At Meta, we are constantly innovating and experimenting by building and shipping many different products, […]

CWChris Wiltz·October 17, 2023SRE & Ops
SRE & Ops — Meta contributes new features to Python 3.12

Meta contributes new features to Python 3.12

Python 3.12 is out! It includes new features and performance improvements – some contributed by Meta – that we believe will benefit all Python users. We’re sharing details about these new features that we worked closely with the Python community to develop. This week’s release of Python 3.12 marks a milestone in our efforts to […]

CWChris Wiltz·October 5, 2023SRE & Ops
Security — Meta Quest 2: Defense through offense

Meta Quest 2: Defense through offense

Meta’s Native Assurance team regularly performs manual code reviews as part of our ongoing commitment to improve the security posture of Meta’s products. In 2021, we discovered a vulnerability in the Meta Quest 2’s Android-based OS that never made it to production but helped us find new ways to improve the security of Meta Quest […]

CWChris Wiltz·September 12, 2023Security
AI & ML — Using Chakra execution traces for benchmarking and network performance optimization

Using Chakra execution traces for benchmarking and network performance optimization

Meta presents Chakra execution traces, an open graph-based representation of AI/ML workload execution, laying the foundation for benchmarking and network performance optimization. Chakra execution traces represent key operations, such as compute, memory, and communication, data and control dependencies, timing, and resource constraints. In collaboration with MLCommons, we are seeking industry-wide

CWChris Wiltz·September 7, 2023AI & ML
SRE & Ops — Arcadia: An end-to-end AI system performance simulator

Arcadia: An end-to-end AI system performance simulator

We’re introducing Arcadia, Meta’s unified system that simulates the compute, memory, and network performance of AI training clusters. Extracting maximum performance from an AI cluster and increasing overall efficiency warrants a multi-input system that accounts for various hardware and software parameters across compute, storage, and network collectively. Arcadia gives Meta’s researchers and engin

CWChris Wiltz·September 7, 2023SRE & Ops
Career & Teams — Threads: The inside story of Meta’s newest social app

Threads: The inside story of Meta’s newest social app

Earlier this year, a small team of engineers at Meta started working on an idea for a new app. It would have all the features people expect from a text-based conversations app, but with one very key, distinctive goal – being an app that would allow people to share their content across multiple platforms. We […]

CWChris Wiltz·September 7, 2023Career & Teams
Career & Teams — What is it like to write code at Meta?

What is it like to write code at Meta?

Ever wonder what it’s like to write code at Meta’s scale? On the latest episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) sits down with Dustin Shahidehpour and Katherine Zak, two software engineers at Meta, to talk about their careers and what it’s really like to ship code at Meta. Why does Meta have […]

CWChris Wiltz·September 5, 2023Career & Teams
SRE & Ops — Scheduling Jupyter Notebooks at Meta

Scheduling Jupyter Notebooks at Meta

At Meta, Bento is our internal Jupyter notebooks platform that is leveraged by many internal users. Notebooks are also being used widely for creating reports and workflows (for example, performing data ETL) that need to be repeated at certain intervals. Users with such notebooks would have to remember to manually run their notebooks at the […]

CWChris Wiltz·August 29, 2023SRE & Ops
SRE & Ops — Introducing Immortal Objects for Python

Introducing Immortal Objects for Python

Instagram has introduced Immortal Objects – PEP-683 – to Python. Now, objects can bypass reference count checks and live throughout the entire execution of the runtime, unlocking exciting avenues for true parallelism. At Meta, we use Python (Django) for our frontend server within Instagram. To handle parallelism, we rely on a multi-process architecture along with […]

CWChris Wiltz·August 15, 2023SRE & Ops
AI & ML — Scaling the Instagram Explore recommendations system

Scaling the Instagram Explore recommendations system

Explore is one of the largest recommendation systems on Instagram. We leverage machine learning to make sure people are always seeing content that is the most interesting and relevant to them. Using more advanced machine learning models, like Two Towers neural networks, we’ve been able to make the Explore recommendation system even more scalable and […]

CWChris Wiltz·August 9, 2023AI & ML
Security — How Meta is improving password security and preserving privacy

How Meta is improving password security and preserving privacy

Meta is developing new privacy-enhancing technologies (PETs) to innovate and solve problems with less data. These technologies enable teams to build and launch privacy-enhanced products in a way that’s verifiable and safeguards user data. Using state-of-the-art cryptographic techniques, we have developed Private Data Lookup (PDL) that allows users to privately query a server-side data set. […] Rea

MEMeta Engineering·August 8, 2023Security
SRE & Ops — Fixit 2: Meta’s next-generation auto-fixing linter

Fixit 2: Meta’s next-generation auto-fixing linter

Fixit is dead! Long live Fixit 2 – the latest version of our open-source auto-fixing linter. Fixit 2 allows developers to efficiently build custom lint rules and perform auto-fixes for their codebases. Fixit 2 is available today on PyPI. Python is one of the most popular languages in use at Meta. Meta’s production engineers (PEs) […]

CWChris Wiltz·August 7, 2023SRE & Ops
SRE & Ops — Using short-lived certificates to protect TLS secrets

Using short-lived certificates to protect TLS secrets

Short-lived certificates (SLCs) are part of our latest efforts to further secure our Transport Layer Security (TLS) private keys on our edge networks. SLCs have a very short exposure compared to traditional certificates and lower the chances of a compromised private key being abused. Implementing SLCs has required us to address tradeoffs between operability and […]

CWChris Wiltz·August 7, 2023SRE & Ops
Frontend — Bringing HDR video to Reels

Bringing HDR video to Reels

Meta has made it possible for people to upload high dynamic range (HDR) videos from their phone’s camera roll to Reels on Facebook and Instagram. To show standard dynamic range (SDR) UI elements and overlays legibly on top of HDR video, we render them at a brightness level comparable to the video itself. We solved […]

MEMeta Engineering·July 17, 2023Frontend
Networking — Meta’s Evenstar is transitioning to OCP to accelerate open RAN adoption

Meta’s Evenstar is transitioning to OCP to accelerate open RAN adoption

Meta is transferring its IP for Evenstar, a program to accelerate the adoption of open RAN technologies, to the Open Compute Project (OCP). Meta will contribute Evenstar’s radio unit design to OCP, giving the telecom industry its first open, white box radio unit solution. The TIP Open RAN community will leverage the Evenstar radio unit […]

CWChris Wiltz·June 29, 2023Networking
SRE & Ops — Meta developer tools: Working at scale

Meta developer tools: Working at scale

Every day, thousands of developers at Meta are working in repositories with millions of files. Those developers need tools that help them at every stage of the workflow while working at extreme scale. In this article we’ll go through a few of the tools in the development process. And, as an added bonus, those we […]

CWChris Wiltz·June 27, 2023SRE & Ops
Networking — Bombyx is being licensed for product development

Bombyx is being licensed for product development

When we first conceived of our aerial fiber deployment solution, Bombyx (the Latin name for a silk moth), we imagined a robot weaving strands of fiber-optic cables over powerlines, helping human workers quickly connect communities even in very rural or remote locations. Now, after years of successful research, Bombyx is taking the next steps in […]

CWChris Wiltz·May 22, 2023Networking
SRE & Ops — Building and deploying MySQL Raft at Meta

Building and deploying MySQL Raft at Meta

We’re rolling out MySQL Raft with the aim to eventually replace our current MySQL semisynchronous databases. The biggest win of MySQL Raft was simplification of the operation and making MySQL servers take care of promotions and membership. This gave the provable safety of Raft and reduced significant operational pain. Making MySQL server a true […]

MEMeta Engineering·May 16, 2023SRE & Ops
Security — The malware threat landscape: NodeStealer, DuckTail, and more

The malware threat landscape: NodeStealer, DuckTail, and more

We’re sharing our latest threat research and technical analysis into persistent malware campaigns targeting businesses across the internet, including threat indicators to help raise our industry’s collective defenses across the internet. These malware families – including Ducktail, NodeStealer and newer malware posing as ChatGPT and other similar tools– targeted people through malicious browser ex

MEMeta Engineering·May 3, 2023Security
Networking — A fine-grained network traffic analysis with Millisampler

A fine-grained network traffic analysis with Millisampler

What the research is: Millisampler is one of Meta’s latest characterization tools and allows us to observe, characterize, and debug network performance at high-granularity timescales efficiently. This lightweight network traffic characterization tool for continual monitoring operates at fine, configurable timescales. It collects time series of ingress and egress traffic volumes, number of active f

MEMeta Engineering·April 17, 2023Networking
Frontend — Deploying key transparency at WhatsApp

Deploying key transparency at WhatsApp

WhatsApp has launched a new cryptographic security feature to automatically verify a secured connection based on key transparency. The feature requires no additional actions or steps from users and helps ensure that a conversation is secure. Key transparency solutions help strengthen the guarantee that end-to-end encryption provides to private, personal messaging applications in a transparent

CWChris Wiltz·April 13, 2023Frontend
Frontend — How Device Verification protects your WhatsApp account

How Device Verification protects your WhatsApp account

WhatsApp has launched a new security feature that further helps prevent attackers from using vectors like on-device malware. This security feature, called Device Verification, requires no action or additional steps from users and helps protect your account. This feature is part of our broader work to increase security for our users from the growing threat […]

CWChris Wiltz·April 13, 2023Frontend
Frontend — Why xHE-AAC is being embraced at Meta

Why xHE-AAC is being embraced at Meta

We’re sharing how Meta delivers high-quality audio at scale with the xHE-AAC audio codec. xHE-AAC has already been deployed on Facebook and Instagram to provide enhanced audio for features like Reels and Stories. At Meta, we serve every media use case imaginable for billions of people across the world — from short-form, user-generated content, such […]

CWChris Wiltz·April 11, 2023Frontend
SRE & Ops — Build faster with Buck2: Our open source build system

Build faster with Buck2: Our open source build system

Buck2, our new open source, large-scale build system, is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient. In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1. Buck2, Meta’s open […]

CWChris Wiltz·April 6, 2023SRE & Ops
SRE & Ops — Introducing Velox: An open source unified execution engine

Introducing Velox: An open source unified execution engine

Meta is introducing Velox, an open source unified execution engine aimed at accelerating data management systems and streamlining their development. Velox is under active development. Experimental results from our paper published at the International Conference on Very Large Data Bases (VLDB) 2022 show how Velox improves efficiency and consistency in data management systems. Velox helps […] Read M

CWChris Wiltz·March 9, 2023SRE & Ops
Engineering — Meta’s head of AR hardware on the future of AR

Meta’s head of AR hardware on the future of AR

While VR headsets have been with us for at least a decade, AR hardware barely exists today; indeed, the very components that will comprise the hardware scarcely exist, making it a truly zero-to-one innovation challenge. Meta’s Head of AR Glasses Hardware, Caitlin Kalinowski is helping to lead that charge. Kalinowski hails from Portsmouth, NH and studied mechanical […]

MEMeta Engineering·February 24, 2023Engineering
Engineering — How Meta brought AV1 to Reels

How Meta brought AV1 to Reels

We’re sharing how we’re enabling production and delivery of AV1 for Facebook Reels and Instagram Reels. We believe AV1 is the most viable codec for Meta for the coming years. It offers higher quality at a much lower bit rate compared with previous generations of video codecs. Meta has worked closely with the open source […]

CWChris Wiltz·February 21, 2023Engineering
Engineering — Inside Meta’s first smart glasses

Inside Meta’s first smart glasses

What’s new: Meta is sharing the inside story of how it developed the Ray-Ban Stories smart glasses. Why it matters: Creating Ray-Ban Stories meant Meta’s engineers had to take on new challenges to build smart glasses that married complex engineering dynamics. How do you make something that features cameras, microphones, audio, and touch controls, all […]

CWChris Wiltz·February 16, 2023Engineering
SRE & Ops — Building a cross-platform runtime for AR

Building a cross-platform runtime for AR

Meta’s augmented reality (AR) platform is one of the largest in the world, helping the billions of people on Meta’s apps experience AR every day and giving hundreds of thousands of creators a means to express themselves Meta’s AR tools are unique because they can be used on a wide variety of devices — from […]

MEMeta Engineering·February 13, 2023SRE & Ops
Engineering — Improving Meta’s global maps

Improving Meta’s global maps

A lot has changed since the initial launch of our basemap in late 2020. We’re Meta now, but our mission remains the same: Giving people the power to build community and bring the world closer together. Across Meta, our family of applications (Facebook, Instagram, WhatsApp, among others) are using our basemap to connect people through […]

CWChris Wiltz·February 7, 2023Engineering
SRE & Ops — The evolution of Facebook’s iOS app architecture

The evolution of Facebook’s iOS app architecture

Facebook for iOS (FBiOS) is the oldest mobile codebase at Meta. Since the app was rewritten in 2012, it has been worked on by thousands of engineers and shipped to billions of users, and it can support hundreds of engineers iterating on it at a time. After years of iteration, the Facebook codebase does not […]

CWChris Wiltz·February 6, 2023SRE & Ops
SRE & Ops — Asynchronous computing at Meta: Overview and learnings

Asynchronous computing at Meta: Overview and learnings

We’ve made architecture changes to Meta’s event driven asynchronous computing platform that have enabled easy integration with multiple event-sources. We’re sharing our learnings from handling various workloads and how to tackle trade offs made with certain design choices in building the platform. Asynchronous computing is a paradigm where the user does not expect a workload […]

CWChris Wiltz·January 31, 2023SRE & Ops
Networking — Watch Meta’s engineers discuss optimizing large-scale networks

Watch Meta’s engineers discuss optimizing large-scale networks

Managing network solutions amidst a growing scale inherently brings challenges around performance, deployment, and operational complexities. At Meta, we’ve found that these challenges broadly fall into three themes: 1.) Data center networking: Over the past decade, on the physical front, we have seen a rise in vendor-specific hardware that comes with heterogeneous feature and […]

CWChris Wiltz·January 27, 2023Networking
SRE & Ops — Tulip: Modernizing Meta’s data platform

Tulip: Modernizing Meta’s data platform

The technical journey discusses the motivations, challenges, and technical solutions employed for warehouse schematization, especially a change to the wire serialization format employed in Meta’s data platform for data interchange related to Warehouse Analytics Logging. Here, we discuss the engineering, scaling, and nontechnical challenges of modernizing Meta’s exabyte-scale data platform by migra

CWChris Wiltz·January 26, 2023SRE & Ops
SRE & Ops — Open-sourcing Anonymous Credential Service

Open-sourcing Anonymous Credential Service

Meta has open-sourced Anonymous Credential Service (ACS), a highly available multitenant service that allows clients to authenticate in a de-identified manner. ACS enhances privacy and security while also being compute-conscious. By open-sourcing and fostering a community for ACS, we believe we can accelerate the pace of innovation in de-identified authentication. Data minimization — collecting th

CWChris Wiltz·December 12, 2022SRE & Ops
SRE & Ops — Enabling static analysis of SQL queries at Meta

Enabling static analysis of SQL queries at Meta

UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. Infrastructure teams at Meta leverage UPM to build SQL linters, catch user mistakes in SQL code, and perform data lineage analysis […]

MEMeta Engineering·November 30, 2022SRE & Ops