Home/SRE & Ops
Topic

SRE & Ops

470 articles on SRE & Ops.

13,309 articles
SRE & Ops — Zoomer: Powering AI Performance at Meta’s Scale Through Intelligent Debugging and Optimization

Zoomer: Powering AI Performance at Meta’s Scale Through Intelligent Debugging and Optimization

We’re introducing Zoomer, Meta’s comprehensive, automated debugging and optimization platform for AI. Zoomer works across all of our training and inference workloads at Meta and provides deep performance insights that enable energy savings, workflow acceleration, and efficiency gains in our AI infrastructure. Zoomer has delivered training time reductions, and significant QPS improvements, making i

CWChris Wiltz·November 21, 2025SRE & Ops
SRE & Ops — Cloudflare outage on November 18, 2025

Cloudflare outage on November 18, 2025

Cloudflare suffered a service outage on November 18, 2025. The outage was triggered by a bug in generation logic for a Bot Management feature file causing many Cloudflare services to be affected.

MPMatthew PrinceMatthew Prince·November 18, 2025SRE & Ops
SRE & Ops — Open Source Is Good for the Environment

Open Source Is Good for the Environment

Most people have heard of open-source software. But have you heard about open hardware? And did you know open source can have a positive impact on the environment? On this episode of the Meta Tech Podcast, Pascal Hartig sits down with Dharmesh and Lisa to talk about all things open hardware, and Meta’s biggest announcements […]

CWChris Wiltz·November 14, 2025SRE & Ops
SRE & Ops — Disaggregated Scheduled Fabric: Scaling Meta’s AI Journey

Disaggregated Scheduled Fabric: Scaling Meta’s AI Journey

Disaggregated Schedule Fabric (DSF) is Meta’s next-generation network fabric technology for AI training networks that addresses the challenges of existing Clos-based networks. We’re sharing the challenges and innovations surrounding DSF and discussing future directions, including the creation of mega clusters through DSF and non-DSF region interconnectivity, as well as the exploration of alternati

CWChris Wiltz·October 20, 2025SRE & Ops
SRE & Ops — Branching in a Sapling Monorepo

Branching in a Sapling Monorepo

Sapling is a scalable, user-friendly, and open-source source control system that powers Meta’s monorepo. As discussed at the GitMerge 2024 conference session on branching, designing and implementing branching workflows for large monorepos is a challenging problem with multiple tradeoffs between scalability and the developer experience. After the conference, we designed, implemented, and open sourc

CWChris Wiltz·October 16, 2025SRE & Ops
SRE & Ops — 10X Backbone: How Meta Is Scaling Backbone Connectivity for AI

10X Backbone: How Meta Is Scaling Backbone Connectivity for AI

We’re sharing details on our journey to scale Meta’s Backbone network to support the increasing demands of new and existing AI workloads. We’ve developed new technologies and designs to address our 10x scaling needs and applying some of these same principles to help extend our AI clusters between multiple data centers. Meta’s Backbone network is […]

PHPhil Hornshaw·October 16, 2025SRE & Ops
SRE & Ops — Design for Sustainability: New Design Principles for Reducing IT Hardware Emissions

Design for Sustainability: New Design Principles for Reducing IT Hardware Emissions

We’re presenting Design for Sustainability, a set of technical design principles for new designs of IT hardware to reduce emissions and cost through reuse, extending useful life, and optimizing design. At Meta, we’ve been able to significantly reduce the carbon footprint of our data centers by integrating several design strategies such as modularity, reuse, retrofitting, […]

CWChris Wiltz·October 14, 2025SRE & Ops
SRE & Ops — OCP Summit 2025: The Open Future of Networking Hardware for AI

OCP Summit 2025: The Open Future of Networking Hardware for AI

At Open Compute Project Summit (OCP) 2025, we’re sharing details about the direction of next-generation network fabrics for our AI training clusters. We’ve expanded our network hardware portfolio and are contributing new disaggregated network platforms to OCP. We look forward to continued collaboration with OCP to open designs for racks, servers, storage boxes, and motherboards […]

CWChris Wiltz·October 13, 2025SRE & Ops
SRE & Ops — Introducing OpenZL: An Open Source Format-Aware Compression Framework

Introducing OpenZL: An Open Source Format-Aware Compression Framework

OpenZL is a new open source data compression framework that offers lossless compression for structured data. OpenZL is designed to offer the performance of a format-specific compressor with the easy maintenance of a single executable binary. You can get started with OpenZL today by visiting our Quick Start guide and the OpenZL GitHub repository. Learn more […]

CWChris Wiltz·October 6, 2025SRE & Ops
SRE & Ops — Cloudflare incident on August 21, 2025

Cloudflare incident on August 21, 2025

On August 21, 2025, an influx of traffic directed toward clients hosted in AWS us-east-1 caused severe congestion on links between Cloudflare and us-east-1. In this post, we explain what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again.

CCloudflare·August 22, 2025SRE & Ops
SRE & Ops — Creating AI agent solutions for warehouse data access and security

Creating AI agent solutions for warehouse data access and security

In this post, we explore the ways we’re evolving Meta’s data warehouse to facilitate productivity and security to serve both human users and AI agents. We detail how we’re developing agents that help users making data access requests to get to the data they need, and that help data owners process requests and maintain security. […]

PHPhil Hornshaw·August 13, 2025SRE & Ops
SRE & Ops — Federation Platform and Privacy Waves: How Meta distributes compliance-related tasks at scale

Federation Platform and Privacy Waves: How Meta distributes compliance-related tasks at scale

We’re exploring Meta’s Federation Platform, a scalable set of tools for managing compliance-related tasks, along with Privacy Waves, our method for batching these tasks and ensuring accountability. Together, the Federation Platform and Privacy Waves create a structured, effective, and sustainable approach to operationalizing privacy work, enabling Meta to safeguard user data for the billions of […

CWChris Wiltz·August 11, 2025SRE & Ops
SRE & Ops — How Meta keeps its AI hardware reliable

How Meta keeps its AI hardware reliable

Hardware faults can have a significant impact on AI training and inference. Silent data corruptions (SDCs), undetected data errors caused by hardware, can be particularly harmful for AI systems that rely on accurate data for training as well as providing useful outputs. We are sharing methodologies we deploy at various scales for detecting SDC across […]

CWChris Wiltz·July 22, 2025SRE & Ops
SRE & Ops — Journey to 1000 models: Scaling Instagram’s recommendation system

Journey to 1000 models: Scaling Instagram’s recommendation system

In this post, we explore how Instagram has successfully scaled its algorithm to include over 1000 ML models without sacrificing recommendation quality or reliability. We delve into the intricacies of managing such a vast array of models, each with its own performance characteristics and product goals. We share insights and lessons learned along the way—from […]

CWChris Wiltz·May 21, 2025SRE & Ops
SRE & Ops — Meta’s Full-stack HHVM optimizations for GenAI

Meta’s Full-stack HHVM optimizations for GenAI

As Meta has launched new, innovative products leveraging generative AI (GenAI), we need to make sure the underlying infrastructure components evolve along with it. Applying infrastructure knowledge and optimizations have allowed us to adapt to changing product requirements, delivering a better product along the way. Ultimately, our infrastructure systems need to balance our need to […]

CWChris Wiltz·May 20, 2025SRE & Ops
SRE & Ops — Introducing Pyrefly: A new type checker and IDE experience for Python

Introducing Pyrefly: A new type checker and IDE experience for Python

Today we are announcing an alpha version of Pyrefly, an open source Python type checker and IDE extension crafted in Rust. Pyrefly is a static type checker that analyzes Python code to ensure type consistency and help you catch errors throughout your codebase before your code runs. It also supports IDE integration and CLI usage […]

CWChris Wiltz·May 15, 2025SRE & Ops
SRE & Ops — Enhancing the Python ecosystem with type checking and free threading

Enhancing the Python ecosystem with type checking and free threading

Meta and Quansight have improved key libraries in the Python Ecosystem. There is plenty more to do and we invite the community to help with our efforts. We’ll look at two key efforts in Python’s packaging ecosystem to make packages faster and easier to use: 🚀 Unlock performance wins for developers through free-threaded Python – […]

CWChris Wiltz·May 5, 2025SRE & Ops
SRE & Ops — Expanding observability on Vercel

Expanding observability on Vercel

The Vercel Marketplace adds new native integrations from Sentry, Checkly, and Dash0. Use the tools you already trust to monitor, measure, and debug your apps with integrated billing, single sign-on, and access to provider dashboards from Vercel.

VVercel·April 8, 2025SRE & Ops
SRE & Ops — Mobile GraphQL at Meta in 2025

Mobile GraphQL at Meta in 2025

Mobile GraphQL is a framework used at Meta for fetching data in mobile applications using GraphQL, a strongly-typed, declarative query language. At Meta it handles data fetching for apps like Facebook and Instagram. Sabrina, a software engineer on Meta’s Mobile GraphQL Platform Team, joins Pascal Hartig on the Meta Tech podcast to discuss the evolution […]

CWChris Wiltz·March 31, 2025SRE & Ops
SRE & Ops — Impromptu disaster recovery

Impromptu disaster recovery

Background im-promp-tu ( im-ˈpräm(p)-(ˌ)tü ) made, done, or formed on or as if on the spur of the moment: improvised composed or uttered without previous preparation: extemporaneous Merriam-Webster On March 18th, 2025, I thought I would look into self-hosted project management solutions — something kanban-y, but.. better? This one does not spark joy. After discovering that Teamhood was awesome (an

AWAmos WengerAmos Wenger·March 28, 2025SRE & Ops
SRE & Ops — Cloudflare incident on March 21, 2025

Cloudflare incident on March 21, 2025

On March 21, 2025, multiple Cloudflare services, including R2 object storage experienced an elevated rate of error responses. Here’s what caused the incident, the impact, and how we are making sure it doesn’t happen again.

PPhillipPhillip·March 25, 2025SRE & Ops
SRE & Ops — A case for QLC SSDs in the data center

A case for QLC SSDs in the data center

The growth of data and need for increased power efficiency are leading to innovative storage solutions. HDDs have been growing in density, but not performance, and TLC flash remains at a price point that is restrictive for scaling. QLC technology addresses these challenges by forming a middle tier between HDDs and TLC SSDs. QLC […]

CWChris Wiltz·March 4, 2025SRE & Ops
SRE & Ops — Cloudflare incident on February 6, 2025

Cloudflare incident on February 6, 2025

On Thursday, February 6, 2025, we experienced an outage with our object storage service (R2) and products that rely on it. Here's what happened and what we're doing to fix this going forward.

SJSilverlock, JavierSilverlock, Javier·February 7, 2025SRE & Ops
SRE & Ops — Indexing code at scale with Glean

Indexing code at scale with Glean

We’re sharing details about Glean, Meta’s open source system for collecting, deriving, and working with facts about source code. In this blog post we’ll talk about why a system like Glean is important, explain the rationale for Glean’s design, and run through some of the ways we’re using Glean to supercharge our developer tooling at […]

CWChris Wiltz·December 19, 2024SRE & Ops
SRE & Ops — Cloudflare incident on November 14, 2024, resulting in lost logs

Cloudflare incident on November 14, 2024, resulting in lost logs

On November 14, 2024, Cloudflare experienced a Cloudflare Logs outage, impacting the majority of customers using these products. During the ~3.5 hours that these services were impacted, about 55% of the logs we normally send to customers were not sent and were lost. The details of what went wrong and why are interesting both for customers and practitioners.

JTJamie, Tom WalwynJamie, Tom Walwyn·November 26, 2024SRE & Ops
SRE & Ops — Sequence learning: A paradigm shift for personalized ads recommendations

Sequence learning: A paradigm shift for personalized ads recommendations

AI plays a fundamental role in creating valuable connections between people and advertisers within Meta’s family of apps. Meta’s ad recommendation engine, powered by deep learning recommendation models (DLRMs), has been instrumental in delivering personalized ads to people. Key to this success was incorporating thousands of human-engineered signals or features in the DLRM-based recommendation syst

CWChris Wiltz·November 19, 2024SRE & Ops
SRE & Ops — There’s No Such Thing as a Free Lunch!

There’s No Such Thing as a Free Lunch!

Incident Management takes time Incidents need responders that are trained and experienced. At Slack, training is a foundation of our incident management program. Self-service training and live courses based mainly on prepared content are one piece of the puzzle, but there can be a missing piece in many organizations. How can staff get practical experience…

SNScott Nelson WindelsScott Nelson Windels·November 14, 2024SRE & Ops
SRE & Ops — Fearless SSH: short-lived certificates bring Zero Trust to infrastructure

Fearless SSH: short-lived certificates bring Zero Trust to infrastructure

Access for Infrastructure, BastionZero’s integration into Cloudflare One, will enable organizations to apply Zero Trust controls to their servers, databases, Kubernetes clusters, and more. Today we’re announcing short-lived SSH access as the first available feature of this integration.

GAGoldbe, Ann Ming SamborskiGoldbe, Ann Ming Samborski·October 23, 2024SRE & Ops
SRE & Ops — OCP Summit 2024: The open future of networking hardware for AI

OCP Summit 2024: The open future of networking hardware for AI

At Open Compute Project Summit (OCP) 2024, we’re sharing details about our next-generation network fabric for our AI training clusters. We’ve expanded our network hardware portfolio and are contributing two new disaggregated network fabrics and a new NIC to OCP. We look forward to continued collaboration with OCP to open designs for racks, servers, storage […]

CWChris Wiltz·October 15, 2024SRE & Ops
SRE & Ops — Meta’s open AI hardware vision

Meta’s open AI hardware vision

At the Open Compute Project (OCP) Global Summit 2024, we’re showcasing our latest open AI hardware designs with the OCP community. These innovations include a new AI platform, cutting-edge open rack designs, and advanced network fabrics and components. By sharing our designs, we hope to inspire collaboration and foster innovation. If you’re passionate about building […]

CWChris Wiltz·October 15, 2024SRE & Ops
SRE & Ops — Leveraging Kubernetes virtual machines at Cloudflare with KubeVirt

Leveraging Kubernetes virtual machines at Cloudflare with KubeVirt

The Kubernetes team runs several multi-tenant clusters across Cloudflare’s core data centers. When multi-tenant cluster isolation is too limiting for an application, we use KubeVirt. KubeVirt is a cloud-native solution that enables our developers to run virtual machines alongside containers.

JCJustin CichraJustin Cichra·October 8, 2024SRE & Ops
SRE & Ops — Cloudflare incident on September 17, 2024

Cloudflare incident on September 17, 2024

On September 17, 2024, during planned routine maintenance, Cloudflare stopped announcing 15 IPv4 prefixes, affecting some Business plan websites for approximately one hour. During this time, IPv4 traffic for these customers would not have reached Cloudflare and users attempting to connect to websites using addresses within those prefixes would have received errors.

JAJoe AbleyJoe Abley·September 20, 2024SRE & Ops
SRE & Ops — Advancing Our Chef Infrastructure

Advancing Our Chef Infrastructure

At Slack, we manage tens of thousands of EC2 instances that host a variety of services, including our Vitess databases, Kubernetes workers, and various components of the Slack application. The majority of these instances run on some version of Ubuntu, while a portion operates on Amazon Linux. With such a vast infrastructure, the critical question…

AGArchie Gunasekara·September 17, 2024SRE & Ops
SRE & Ops — Simulator-based reinforcement learning for data center cooling optimization

Simulator-based reinforcement learning for data center cooling optimization

We’re sharing more about the role that reinforcement learning plays in helping us optimize our data centers’ environmental controls. Our reinforcement learning-based approach has helped us reduce energy consumption and water usage across various weather conditions in our data centers. Meta is revamping its new data center design to optimize for artificial intelligence and the […]

CWChris Wiltz·September 10, 2024SRE & Ops
SRE & Ops — RETINAS: Real-Time Infrastructure Accounting for Sustainability

RETINAS: Real-Time Infrastructure Accounting for Sustainability

We are introducing a new metric— real-time server fleet utilization effectiveness —as part of the RETINAS initiative to help reduce emissions and achieve net zero emissions across our value chain in 2030. This new metric allows us to measure server resource usage (e.g., compute, storage) and efficiency in our large-scale data center server fleet in […]

CWChris Wiltz·August 26, 2024SRE & Ops