Home/SRE & Ops
Topic

SRE & Ops

470 articles on SRE & Ops.

13,309 articles
SRE & Ops — Using short-lived certificates to protect TLS secrets

Using short-lived certificates to protect TLS secrets

Short-lived certificates (SLCs) are part of our latest efforts to further secure our Transport Layer Security (TLS) private keys on our edge networks. SLCs have a very short exposure compared to traditional certificates and lower the chances of a compromised private key being abused. Implementing SLCs has required us to address tradeoffs between operability and […]

CWChris Wiltz·August 7, 2023SRE & Ops
SRE & Ops — Moving from a product to a service mindset

Moving from a product to a service mindset

Thanks to DevOps, cloud computing and other industry trends, many organizations are shifting from a product mindset to a service mindset. Here’s how you can implement a service-led strategy.

CRChris ReddingtonChris Reddington·July 13, 2023SRE & Ops
SRE & Ops — Meta developer tools: Working at scale

Meta developer tools: Working at scale

Every day, thousands of developers at Meta are working in repositories with millions of files. Those developers need tools that help them at every stage of the workflow while working at extreme scale. In this article we’ll go through a few of the tools in the development process. And, as an added bonus, those we […]

CWChris Wiltz·June 27, 2023SRE & Ops
SRE & Ops — Analyzing Volatile Memory on a Google Kubernetes Engine Node

Analyzing Volatile Memory on a Google Kubernetes Engine Node

TL:DR At Spotify, we run containerized workloads in production across our entire organization in five regions where our main production workloads are in Google Kubernetes Engine (GKE) on Google Cloud Platform (GCP). If we detect suspicious behavior in our workloads, we need to be able to quickly analyze it and determine if something malicious has happened. Today we leverage commercial solutions to

MHMarcus Hallberg, Security Engineer·June 1, 2023SRE & Ops
SRE & Ops — Building and deploying MySQL Raft at Meta

Building and deploying MySQL Raft at Meta

We’re rolling out MySQL Raft with the aim to eventually replace our current MySQL semisynchronous databases. The biggest win of MySQL Raft was simplification of the operation and making MySQL servers take care of promotions and membership. This gave the provable safety of Raft and reduced significant operational pain. Making MySQL server a true […]

MEMeta Engineering·May 16, 2023SRE & Ops
SRE & Ops — Announcing GitHub Actions Deployment Protection Rules, now in public beta

Announcing GitHub Actions Deployment Protection Rules, now in public beta

Create and share your own deployment protection rules, or use the rules from our great partners, like Datadog, Honeycomb, New Relic, NodeSource, Sentry, and ServiceNow, to control your deployments with more confidence. And the API is open for the community to build their own rules to make GitHub Enterprise Cloud even better.

BRBrian RandellBrian Randell·April 20, 2023SRE & Ops
SRE & Ops — Build faster with Buck2: Our open source build system

Build faster with Buck2: Our open source build system

Buck2, our new open source, large-scale build system, is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient. In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1. Buck2, Meta’s open […]

CWChris Wiltz·April 6, 2023SRE & Ops
SRE & Ops — Building organization-wide governance and re-use for CI/CD and automation with GitHub Actions

Building organization-wide governance and re-use for CI/CD and automation with GitHub Actions

Many of us are aware of the benefits that a strong focus on automation can bring, particularly in our development workflow and DevOps lifecycle. But silos across businesses can lead to duplication of effort, and potential to lose out on best practices. In this post, we’ll explore how CI/CD can be shared across your entire organization alongside policies, for a well-governed experience with GitHub

CRChris ReddingtonChris Reddington·April 5, 2023SRE & Ops
SRE & Ops — Technology Lifecycle

Technology Lifecycle

This blog post discusses the strategies that Slack uses to manage the lifecycle (development, support, and eventual retirement) of infrastructure projects, through the lens of the migration through three successive internal “platform” offerings. Our challenges Circa 2020, our Cloud Engineering team (now evolved into multiple teams responsible for narrower aspects) was responsible for managing our…

TBTricia Bogen·March 21, 2023SRE & Ops
SRE & Ops — Introducing Velox: An open source unified execution engine

Introducing Velox: An open source unified execution engine

Meta is introducing Velox, an open source unified execution engine aimed at accelerating data management systems and streamlining their development. Velox is under active development. Experimental results from our paper published at the International Conference on Very Large Data Bases (VLDB) 2022 show how Velox improves efficiency and consistency in data management systems. Velox helps […] Read M

CWChris Wiltz·March 9, 2023SRE & Ops
SRE & Ops — How Cloudflare runs Prometheus at scale

How Cloudflare runs Prometheus at scale

Here at Cloudflare we run over 900 instances of Prometheus with a total of around 4.9 billion time series. Operating such a large Prometheus deployment doesn’t come without challenges . In this blog post we’ll cover some of the issues we hit and how we solved them

LLukaszLukasz·March 3, 2023SRE & Ops
SRE & Ops — GitHub Actions Importer is now generally available

GitHub Actions Importer is now generally available

We’re excited to announce the general availability of GitHub Actions Importer. GitHub Actions Importer helps you plan, forecast, and automate migrations from Azure DevOps, CircleCI, GitLab, Jenkins, and Travis CI…

DGDawit GebregziabherDawit Gebregziabher·March 1, 2023SRE & Ops
SRE & Ops — Building a cross-platform runtime for AR

Building a cross-platform runtime for AR

Meta’s augmented reality (AR) platform is one of the largest in the world, helping the billions of people on Meta’s apps experience AR every day and giving hundreds of thousands of creators a means to express themselves Meta’s AR tools are unique because they can be used on a wide variety of devices — from […]

MEMeta Engineering·February 13, 2023SRE & Ops
SRE & Ops — The evolution of Facebook’s iOS app architecture

The evolution of Facebook’s iOS app architecture

Facebook for iOS (FBiOS) is the oldest mobile codebase at Meta. Since the app was rewritten in 2012, it has been worked on by thousands of engineers and shipped to billions of users, and it can support hundreds of engineers iterating on it at a time. After years of iteration, the Facebook codebase does not […]

CWChris Wiltz·February 6, 2023SRE & Ops
SRE & Ops — Enabling branch deployments through IssueOps with GitHub Actions

Enabling branch deployments through IssueOps with GitHub Actions

What if developers want to leverage branch deployments but don’t have a full ChatOps stack integrated with their repositories? We wanted to set out to find a way for all developers to be able to take advantage of branch deployments with ease, right from their GitHub repository, and so the branch-deploy Action was born!

GBGrant BirkinbineGrant Birkinbine·February 2, 2023SRE & Ops
SRE & Ops — Incident Report: Spotify Outage on January 14, 2023

Incident Report: Spotify Outage on January 14, 2023

On January 14, between 00:15 UTC and 03:45 UTC, Spotify suffered an outage. The impact was small at first and increased over the course of an hour until most functionality (including playback) was not working. Spotify engineers were first notified of the problem at 00:40 UTC, and our incident response team was immediately assembled. Due to the nature of the incident, triage took longer than one wo

SESpotify Engineering·February 1, 2023SRE & Ops
SRE & Ops — Asynchronous computing at Meta: Overview and learnings

Asynchronous computing at Meta: Overview and learnings

We’ve made architecture changes to Meta’s event driven asynchronous computing platform that have enabled easy integration with multiple event-sources. We’re sharing our learnings from handling various workloads and how to tackle trade offs made with certain design choices in building the platform. Asynchronous computing is a paradigm where the user does not expect a workload […]

CWChris Wiltz·January 31, 2023SRE & Ops
SRE & Ops — Tulip: Modernizing Meta’s data platform

Tulip: Modernizing Meta’s data platform

The technical journey discusses the motivations, challenges, and technical solutions employed for warehouse schematization, especially a change to the wire serialization format employed in Meta’s data platform for data interchange related to Warehouse Analytics Logging. Here, we discuss the engineering, scaling, and nontechnical challenges of modernizing Meta’s exabyte-scale data platform by migra

CWChris Wiltz·January 26, 2023SRE & Ops
SRE & Ops — Cloudflare incident on January 24, 2023

Cloudflare incident on January 24, 2023

Several Cloudflare services became unavailable for 121 minutes on January 24th, 2023 due to an error releasing code that manages service tokens. The incident degraded a wide range of Cloudflare products

KSKenny, SamKenny, Sam·January 25, 2023SRE & Ops
SRE & Ops — Open-sourcing Anonymous Credential Service

Open-sourcing Anonymous Credential Service

Meta has open-sourced Anonymous Credential Service (ACS), a highly available multitenant service that allows clients to authenticate in a de-identified manner. ACS enhances privacy and security while also being compute-conscious. By open-sourcing and fostering a community for ACS, we believe we can accelerate the pace of innovation in de-identified authentication. Data minimization — collecting th

CWChris Wiltz·December 12, 2022SRE & Ops
SRE & Ops — Enabling static analysis of SQL queries at Meta

Enabling static analysis of SQL queries at Meta

UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. Infrastructure teams at Meta leverage UPM to build SQL linters, catch user mistakes in SQL code, and perform data lineage analysis […]

MEMeta Engineering·November 30, 2022SRE & Ops
SRE & Ops — Retrofitting null-safety onto Java at Meta

Retrofitting null-safety onto Java at Meta

We developed a new static analysis tool called Nullsafe that is used at Meta to detect NullPointerException (NPE) errors in Java code. Interoperability with legacy code and gradual deployment model were key to Nullsafe’s wide adoption and allowed us to recover some null-safety properties in the context of an otherwise null-unsafe language in a multimillion-line […]

CWChris Wiltz·November 22, 2022SRE & Ops
SRE & Ops — Sapling: Source control that’s user-friendly and scalable

Sapling: Source control that’s user-friendly and scalable

Sapling is a new Git-compatible source control client. Sapling emphasizes usability while also scaling to the largest repositories in the world. ReviewStack is a demonstration code review UI for GitHub pull requests that integrates with Sapling to make reviewing stacks of commits easy. You can get started using Sapling today. Source control is one of […]

CWChris Wiltz·November 15, 2022SRE & Ops
SRE & Ops — Tulip: Schematizing Meta’s data platform

Tulip: Schematizing Meta’s data platform

We’re sharing Tulip, a binary serialization protocol supporting schema evolution. Tulip assists with data schematization by addressing protocol reliability and other issues simultaneously. It replaces multiple legacy formats used in Meta’s data platform and has achieved significant performance and efficiency gains. There are numerous heterogeneous services, such as warehouse data storage and vario

CWChris Wiltz·November 9, 2022SRE & Ops
SRE & Ops — Improving Instagram notification management with machine learning and causal inference

Improving Instagram notification management with machine learning and causal inference

We’re sharing how Meta is applying statistics and machine learning (ML) to improve notification personalization and management on Instagram – particularly on daily digest push notifications. By using causal inference and ML to identify highly active users who are likely to see more content organically, we have been able to reduce the number of notifications […]

CWChris Wiltz·October 31, 2022SRE & Ops
SRE & Ops — How We Use Terraform At Slack

How We Use Terraform At Slack

At Slack, we use Terraform for managing our Infrastructure, which runs on AWS, DigitalOcean, NS1, and GCP. Even though most of our infrastructure is running on AWS, we have chosen to use Terraform as opposed to using an AWS-native service such as CloudFormation so that we can use a single tool across all of our…

AGArchie Gunasekara·October 25, 2022SRE & Ops
SRE & Ops — OCP Summit 2022: Open hardware for AI infrastructure

OCP Summit 2022: Open hardware for AI infrastructure

At OCP Summit 2022, we’re announcing Grand Teton, our next-generation platform for AI at scale that we’ll contribute to the OCP community. We’re also sharing new innovations designed to support data centers as they advance to support new AI technologies: A new, more efficient version of Open Rack. Our Air-Assisted Liquid Cooling (AALC) – design. […]

CWChris Wiltz·October 18, 2022SRE & Ops
SRE & Ops — Scaling data ingestion for machine learning training at Meta

Scaling data ingestion for machine learning training at Meta

Many of Meta’s products, such as search, ads ranking and Marketplace, utilize AI models to continuously improve user experiences. As the performance of hardware we use to support training infrastructure increases, we need to scale our data ingestion infrastructure accordingly to handle workloads more efficiently. GPUs, which are used for training infrastructure, tend to double […]

MEMeta Engineering·September 19, 2022SRE & Ops