Home/SRE & Ops
Topic

SRE & Ops

470 articles on SRE & Ops.

13,309 articles
SRE & Ops — DCPerf: An open source benchmark suite for hyperscale compute applications

DCPerf: An open source benchmark suite for hyperscale compute applications

We are open-sourcing DCPerf, a collection of benchmarks that represents the diverse categories of workloads that run in data center cloud deployments. We hope that DCperf can be used more broadly by academia, the hardware industry, and internet companies to design and evaluate future products. DCPerf is available now on GitHub. Hyperscale and cloud datacenter […]

CWChris Wiltz·August 5, 2024SRE & Ops
SRE & Ops — A recent spate of Internet disruptions

A recent spate of Internet disruptions

Cloudflare Radar is constantly monitoring the Internet for widespread disruptions. Here we examine several recent noteworthy disruptions detected in the first month of Q3, including traffic anomalies observed in Bangladesh, Syria, Pakistan, and Venezuela.

DBDavid BelsonDavid Belson·August 1, 2024SRE & Ops
SRE & Ops — AI Lab: The secrets to keeping machine learning engineers moving fast

AI Lab: The secrets to keeping machine learning engineers moving fast

The key to developer velocity across AI lies in minimizing time to first batch (TTFB) for machine learning (ML) engineers. AI Lab is a pre-production framework used internally at Meta. It allows us to continuously A/B test common ML workflows – enabling proactive improvements and automatically preventing regressions on TTFB. AI Lab prevents TTFB regressions […]

CWChris Wiltz·July 16, 2024SRE & Ops
SRE & Ops — Meta’s approach to machine learning prediction robustness

Meta’s approach to machine learning prediction robustness

Meta’s advertising business leverages large-scale machine learning (ML) recommendation models that power millions of ads recommendations per second across Meta’s family of apps. Maintaining reliability of these ML systems helps ensure the highest level of service and uninterrupted benefit delivery to our users and advertisers. To minimize disruptions and ensure our ML systems are intrinsically

CWChris Wiltz·July 10, 2024SRE & Ops
SRE & Ops — The key to a happy Rust/C++ relationship

The key to a happy Rust/C++ relationship

The history of Rust at Meta goes all the way back to 2016, when we first started using it for source control. Today, it has been widely embraced at Meta and is one of our primary supported server-side languages (along with C++, Python, and Hack). But that doesn’t mean there weren’t any growing pains. Aida […]

CWChris Wiltz·June 25, 2024SRE & Ops
SRE & Ops — Leveraging AI for efficient incident response

Leveraging AI for efficient incident response

We’re sharing how we streamline system reliability investigations using a new AI-assisted root cause analysis system. The system uses a combination of heuristic-based retrieval and large language model-based ranking to speed up root cause identification during investigations. Our testing has shown this new system achieves 42% accuracy in identifying root causes for investigations at their […] Read

CWChris Wiltz·June 24, 2024SRE & Ops
SRE & Ops — PVF: A novel metric for understanding AI systems’ vulnerability against SDCs in model parameters

PVF: A novel metric for understanding AI systems’ vulnerability against SDCs in model parameters

We’re introducing parameter vulnerability factor (PVF), a novel metric for understanding and measuring AI systems’ vulnerability against silent data corruptions (SDCs) in model parameters. PVF can be tailored to different AI models and tasks, adapted to different hardware faults, and even extended to the training phase of AI models. We’re sharing results of our own […]

CWChris Wiltz·June 19, 2024SRE & Ops
SRE & Ops — Advanced Rollout Techniques: Custom Strategies for Stateful Apps in Kubernetes

Advanced Rollout Techniques: Custom Strategies for Stateful Apps in Kubernetes

In a previous blog post—A Simple Kubernetes Admission Webhook—I discussed the process of creating a Kubernetes webhook without relying on Kubebuilder. At Slack, we use this webhook for various tasks, like helping us support long-lived Pods (see Supporting Long-Lived Pods), and today, I delve once more into the topic of long-lived Pods, focusing on our…

CLClément Labbe·June 15, 2024SRE & Ops
SRE & Ops — How Meta trains large language models at scale

How Meta trains large language models at scale

As we continue to focus our AI research and development on solving increasingly complex problems, one of the most significant and challenging shifts we’ve experienced is the sheer scale of computation required to train large language models (LLMs). Traditionally, our AI model training has involved a training massive number of models that required a comparatively […]

CWChris Wiltz·June 12, 2024SRE & Ops
SRE & Ops — Maintaining large-scale AI capacity at Meta

Maintaining large-scale AI capacity at Meta

Meta is currently operating many data centers with GPU training clusters across the world. Our data centers are the backbone of our operations, meticulously designed to support the scaling demands of compute and storage. A year ago, however, as the industry reached a critical inflection point due to the rise of artificial intelligence (AI), we […]

CWChris Wiltz·June 12, 2024SRE & Ops
SRE & Ops — Unlocking the power of mixed reality devices with MobileConfig

Unlocking the power of mixed reality devices with MobileConfig

MobileConfig enables developers to centrally manage a mobile app’s configuration parameters in our data centers. Once a parameter value is changed on our central server, billions of app devices automatically fetch and apply the new value without app updates. These remotely managed configuration parameters serve various purposes such as A/B testing, feature rollout, and app […]

CWChris Wiltz·June 11, 2024SRE & Ops
SRE & Ops — Serverless Jupyter Notebooks at Meta

Serverless Jupyter Notebooks at Meta

At Meta, Bento, our internal Jupyter notebooks platform, is a popular tool that allows our engineers to mix code, text, and multimedia in a single document. Use cases run the entire spectrum from what we call “lite” workloads that involve simple prototyping to heavier and more complex machine learning workflows. However, even though the lite […]

CWChris Wiltz·June 10, 2024SRE & Ops
SRE & Ops — Adopting OpenTelemetry for our logging pipeline

Adopting OpenTelemetry for our logging pipeline

Recently, Cloudflare's Observability team undertook an effort to migrate our existing syslog-ng backed logging infrastructure to instead being backed by OpenTelemetry Collectors. In this post, we detail the process that we undertook, and the difficulties we faced along the way

CCloudflare·June 3, 2024SRE & Ops
SRE & Ops — Introducing no-code migrations to Stripe Billing

Introducing no-code migrations to Stripe Billing

Last month we announced a new Billing migration toolkit designed to address your top pain points: engineering resourcing, the risk of disruptive errors, and migration time. Here’s how the migration toolkit addresses each.

DCDivyanti ChauhanDivyanti Chauhan·May 28, 2024SRE & Ops
SRE & Ops — Composable data management at Meta

Composable data management at Meta

In recent years, Meta’s data management systems have evolved into a composable architecture that creates interoperability, promotes reusability, and improves engineering efficiency. We’re sharing how we’ve achieved this, in part, by leveraging Velox, Meta’s open source execution engine, as well as work ahead as we continue to rethink our data management systems. Data is at […]

CWChris Wiltz·May 22, 2024SRE & Ops
SRE & Ops — Behind the scenes of Threads for web

Behind the scenes of Threads for web

When Threads first launched one of the top feature requests was for a web client. In this episode of the Meta Tech Podcast, Pascal Hartig (@passy) sits down with Ally C. and Kevin C., two engineers on the Threads Web Team that delivered the basic version of Threads for web in just under three months. […]

CWChris Wiltz·May 14, 2024SRE & Ops
SRE & Ops — Logarithm: A logging engine for AI training workflows and services

Logarithm: A logging engine for AI training workflows and services

Systems and application logs play a key role in operations, observability, and debugging workflows at Meta. Logarithm is a hosted, serverless, multitenant service, used only internally at Meta, that consumes and indexes these logs and provides an interactive query interface to retrieve and view logs. In this post, we present the design behind Logarithm, and […]

CWChris Wiltz·March 18, 2024SRE & Ops
SRE & Ops — Aligning Velox and Apache Arrow: Towards composable data management

Aligning Velox and Apache Arrow: Towards composable data management

We’ve partnered with Voltron Data and the Arrow community to align and converge Apache Arrow with Velox, Meta’s open source execution engine. Apache Arrow 15 includes three new format layouts developed through this partnership: StringView, ListView, and Run-End-Encoding (REE). This new convergence helps Meta and the larger community build data management systems that are unified, […]

CWChris Wiltz·February 20, 2024SRE & Ops
SRE & Ops — DotSlash: Simplified executable deployment

DotSlash: Simplified executable deployment

We’ve open sourced DotSlash, a tool that makes large executables available in source control with a negligible impact on repository size, thus avoiding I/O-heavy clone operations. With DotSlash, a set of platform-specific executables is replaced with a single script containing descriptors for the supported platforms. DotSlash handles transparently fetching, decompressing, and verifying the appropr

CWChris Wiltz·February 6, 2024SRE & Ops
SRE & Ops — Lazy is the new fast: How Lazy Imports and Cinder accelerate machine learning at Meta

Lazy is the new fast: How Lazy Imports and Cinder accelerate machine learning at Meta

At Meta, the quest for faster model training has yielded an exciting milestone: the adoption of Lazy Imports and the Python Cinder runtime. The outcome? Up to 40 percent time to first batch (TTFB) improvements, along with a 20 percent reduction in Jupyter kernel startup times. This advancement facilitates swifter experimentation capabilities and elevates the […]

CWChris Wiltz·January 18, 2024SRE & Ops
SRE & Ops — The Scary Thing About Automating Deploys

The Scary Thing About Automating Deploys

Most of Slack runs on a monolithic service simply called “The Webapp”. It’s big – hundreds of developers create hundreds of changes every week. Deploying at this scale is a unique challenge. When people talk about continuous deployment, they’re often thinking about deploying to systems as soon as changes are ready. They talk about microservices…

SMSean McIlroy·January 18, 2024SRE & Ops
SRE & Ops — An overview of Cloudflare's logging pipeline

An overview of Cloudflare's logging pipeline

In this post, we’re going to go over what that looks like, how we achieve high availability, and how we meet our Service Level Objectives (SLOs) while shipping close to a million log lines per second

CColinColin·January 8, 2024SRE & Ops
SRE & Ops — How Meta built the infrastructure for Threads

How Meta built the infrastructure for Threads

On July 5, 2023, Meta launched Threads, the newest product in our family of apps, to an unprecedented success that saw it garner over 100 million sign ups in its first five days. A small, nimble team of engineers built Threads over the course of only five months of technical work. While the app’s production […]

CWChris Wiltz·December 19, 2023SRE & Ops
SRE & Ops — AI debugging at Meta with HawkEye

AI debugging at Meta with HawkEye

HawkEye is the powerful toolkit used internally at Meta for monitoring, observability, and debuggability of the end-to-end machine learning (ML) workflow that powers ML-based products. HawkEye supports recommendation and ranking models across several products at Meta. Over the past two years, it has facilitated order of magnitude improvements in the time spent debugging production issues. […] Read

CWChris Wiltz·December 19, 2023SRE & Ops
SRE & Ops — Writing and linting Python at scale

Writing and linting Python at scale

Python plays a big part at Meta. It powers Instagram’s backend and plays an important role in our configuration systems, as well as much of our AI work. Meta even made contributions to Python 3.12, the latest version of Python. On this episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) is joined by Amethyst […]

CWChris Wiltz·November 21, 2023SRE & Ops
SRE & Ops — Cloudflare incident on October 30, 2023

Cloudflare incident on October 30, 2023

Multiple Cloudflare services were unavailable for 37 minutes on October 30, 2023, due to the misconfiguration of a deployment tool used by Workers KV.

SKSilverlock, Kris EvansSilverlock, Kris Evans·November 1, 2023SRE & Ops
SRE & Ops — Automating data removal

Automating data removal

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. SCARF scans production data systems to identify tables or assets that are unused and safely removes them. SCARF avoids tedious manual work and ensures that product data is correctly removed when a product is shut down. This […]

CWChris Wiltz·October 31, 2023SRE & Ops
SRE & Ops — Automating dead code cleanup

Automating dead code cleanup

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing dead code. SCARF combines static and dynamic analysis of programs to detect dead code from both a business and programming language perspective. SCARF automatically creates change requests that delete the dead code identified from the program analysis, minimizing developer costs. […]

CWChris Wiltz·October 24, 2023SRE & Ops
SRE & Ops — 5 Things you didn’t know about Buck2

5 Things you didn’t know about Buck2

Meta has a very large monorepo, with many different programming languages. To optimize build and performance, we developed our own build system called Buck, which was first open-sourced in 2013. Buck2 is the recently open-sourced successor. In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x as fast as Buck1. Below […]

CWChris Wiltz·October 23, 2023SRE & Ops
SRE & Ops — Automating product deprecation

Automating product deprecation

Systematic Code and Asset Removal Framework (SCARF) is Meta’s unused code and data deletion framework. SCARF guides engineers through deprecating a product safely and efficiently via an internal tool. SCARF combines this tooling with automation to reduce load on engineers. At Meta, we are constantly innovating and experimenting by building and shipping many different products, […]

CWChris Wiltz·October 17, 2023SRE & Ops
SRE & Ops — Meta contributes new features to Python 3.12

Meta contributes new features to Python 3.12

Python 3.12 is out! It includes new features and performance improvements – some contributed by Meta – that we believe will benefit all Python users. We’re sharing details about these new features that we worked closely with the Python community to develop. This week’s release of Python 3.12 marks a milestone in our efforts to […]

CWChris Wiltz·October 5, 2023SRE & Ops
SRE & Ops — Traffic anomalies and notifications with Cloudflare Radar

Traffic anomalies and notifications with Cloudflare Radar

Cloudflare Radar now displays country and ASN traffic anomalies in the Outage Center as they are detected, as well as publishing anomaly information via API. We are also launching Radar notifications, enabling users to subscribe to notifications about traffic anomalies

DBDavid BelsonDavid Belson·September 26, 2023SRE & Ops
SRE & Ops — Arcadia: An end-to-end AI system performance simulator

Arcadia: An end-to-end AI system performance simulator

We’re introducing Arcadia, Meta’s unified system that simulates the compute, memory, and network performance of AI training clusters. Extracting maximum performance from an AI cluster and increasing overall efficiency warrants a multi-input system that accounts for various hardware and software parameters across compute, storage, and network collectively. Arcadia gives Meta’s researchers and engin

CWChris Wiltz·September 7, 2023SRE & Ops
SRE & Ops — Scheduling Jupyter Notebooks at Meta

Scheduling Jupyter Notebooks at Meta

At Meta, Bento is our internal Jupyter notebooks platform that is leveraged by many internal users. Notebooks are also being used widely for creating reports and workflows (for example, performing data ETL) that need to be repeated at certain intervals. Users with such notebooks would have to remember to manually run their notebooks at the […]

CWChris Wiltz·August 29, 2023SRE & Ops
SRE & Ops — Slack’s Migration to a Cellular Architecture

Slack’s Migration to a Cellular Architecture

Summary In recent years, cellular architectures have become increasingly popular for large online services as a way to increase redundancy and limit the blast radius of site failures. In pursuit of these goals, we have migrated the most critical user-facing services at Slack from a monolithic to a cell-based architecture over the last 1.5 years.…

SZStephan Zuercher·August 22, 2023SRE & Ops
SRE & Ops — The Swedbank Outage shows that Change Controls don't work - High Scalability -

The Swedbank Outage shows that Change Controls don't work - High Scalability -

This week I’ve been reading through the recent judgment from the Swedish FSA on the Swedbank outage. If you’re unfamiliar with this story, Swedbank had a major outage in April 2022 that was caused by an unapproved change to their IT systems. It temporarily left nearly a million customers with incorrect balances, many of whom were unable to meet payments. After investigation, the regulator found th

BJBruce Johnston·August 16, 2023SRE & Ops
SRE & Ops — Introducing Immortal Objects for Python

Introducing Immortal Objects for Python

Instagram has introduced Immortal Objects – PEP-683 – to Python. Now, objects can bypass reference count checks and live throughout the entire execution of the runtime, unlocking exciting avenues for true parallelism. At Meta, we use Python (Django) for our frontend server within Instagram. To handle parallelism, we rely on a multi-process architecture along with […]

CWChris Wiltz·August 15, 2023SRE & Ops
SRE & Ops — Fixit 2: Meta’s next-generation auto-fixing linter

Fixit 2: Meta’s next-generation auto-fixing linter

Fixit is dead! Long live Fixit 2 – the latest version of our open-source auto-fixing linter. Fixit 2 allows developers to efficiently build custom lint rules and perform auto-fixes for their codebases. Fixit 2 is available today on PyPI. Python is one of the most popular languages in use at Meta. Meta’s production engineers (PEs) […]

CWChris Wiltz·August 7, 2023SRE & Ops